[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-22) (GCC)) #7 SMP Sat Jan 18 21:01:29 EST 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 930774062 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcbe3000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 513 pages/cpu s2064384 r8192 d28672 u4194304 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2818960K/4306352K available (20483K kernel code, 12066K rwdata, 7356K rodata, 4680K init, 23504K bss, 542472K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41388 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x616/0x99a with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar [ 0.001000] ... MAX_LOCKDEP_SUBCLASSES: 8 [ 0.001000] ... MAX_LOCK_DEPTH: 48 [ 0.001000] ... MAX_LOCKDEP_KEYS: 8192 [ 0.001000] ... CLASSHASH_SIZE: 4096 [ 0.001000] ... MAX_LOCKDEP_ENTRIES: 32768 [ 0.001000] ... MAX_LOCKDEP_CHAINS: 65536 [ 0.001000] ... CHAINHASH_SIZE: 32768 [ 0.001000] memory used by lock dependency info: 4149 kB [ 0.001000] per task-struct memory footprint: 2688 bytes [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001026] APIC: Switch to symmetric I/O mode setup [ 0.004226] x2apic enabled [ 0.005019] Switched APIC routing to physical x2apic. [ 0.006040] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009039] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010046] pid_max: default: 32768 minimum: 301 [ 0.011489] LSM: Security Framework initializing [ 0.013054] Yama: becoming mindful. [ 0.014146] SELinux: Initializing. [ 0.016315] *** VALIDATE selinux *** [ 0.029364] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.036000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.037162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.039239] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.040442] *** VALIDATE tmpfs *** [ 0.042000] *** VALIDATE proc *** [ 0.044390] *** VALIDATE cgroup *** [ 0.045025] *** VALIDATE cgroup2 *** [ 0.047083] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.048253] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.049027] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.050040] Spectre V2 : User space: Vulnerable [ 0.051009] Speculative Store Bypass: Vulnerable [ 0.055389] debug: unmapping init [mem 0xffffffff8af03000-0xffffffff8af0afff] [ 0.057775] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.060624] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.061099] ... version: 2 [ 0.062019] ... bit width: 48 [ 0.063024] ... generic registers: 4 [ 0.064032] ... value mask: 0000ffffffffffff [ 0.065054] ... max period: 00007fffffffffff [ 0.066060] ... fixed-purpose events: 3 [ 0.067025] ... event mask: 000000070000000f [ 0.068696] rcu: Hierarchical SRCU implementation. [ 0.075136] smp: Bringing up secondary CPUs ... [ 0.079217] x86: Booting SMP configuration: [ 0.080064] .... node #0, CPUs: #1 [ 0.090010] #2 [ 0.095283] #3 [ 0.101292] smp: Brought up 1 node, 4 CPUs [ 0.102084] smpboot: Max logical packages: 1 [ 0.103014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127508] node 0 deferred pages initialised in 18ms [ 0.128000] pgdatinit0 (35) used greatest stack depth: 14528 bytes left [ 0.141717] devtmpfs: initialized [ 0.146207] x86/mm: Memory block size: 128MB [ 0.165755] gcov: version magic: 0x41383552 [ 0.171622] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.177173] futex hash table entries: 1024 (order: 5, 131072 bytes, vmalloc) [ 0.182172] pinctrl core: initialized pinctrl subsystem [ 0.185530] [ 0.187019] ************************************************************* [ 0.189014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192029] ** ** [ 0.195045] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.201045] ** ** [ 0.204027] ** This means that this kernel is built to expose internal ** [ 0.206050] ** IOMMU data structures, which may compromise security on ** [ 0.210031] ** your system. ** [ 0.212050] ** ** [ 0.216032] ** If you see this message and you are not debugging the ** [ 0.220000] ** kernel, report this immediately to your vendor! ** [ 0.223024] ** ** [ 0.224036] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.226019] ************************************************************* [ 0.242930] NET: Registered protocol family 16 [ 0.245129] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.248224] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.251116] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.259129] cpuidle: using governor menu [ 0.264501] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.267000] PCI: Using configuration type 1 for base access [ 0.268142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.319238] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.320046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.337000] cryptd: max_cpu_qlen set to 1000 [ 0.342216] ACPI: Added _OSI(Module Device) [ 0.343040] ACPI: Added _OSI(Processor Device) [ 0.344014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.345016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.393945] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.404907] ACPI: Interpreter enabled [ 0.405873] ACPI: PM: (supports S0 S3 S4 S5) [ 0.406013] ACPI: Using IOAPIC for interrupt routing [ 0.408182] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.412668] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.523236] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.524066] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.525018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.526458] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.537585] acpiphp: Slot [2] registered [ 0.538496] acpiphp: Slot [5] registered [ 0.539349] acpiphp: Slot [6] registered [ 0.540331] acpiphp: Slot [7] registered [ 0.541417] acpiphp: Slot [8] registered [ 0.542411] acpiphp: Slot [9] registered [ 0.543425] acpiphp: Slot [10] registered [ 0.544391] acpiphp: Slot [3] registered [ 0.545421] acpiphp: Slot [4] registered [ 0.546314] acpiphp: Slot [11] registered [ 0.547321] acpiphp: Slot [12] registered [ 0.548547] acpiphp: Slot [13] registered [ 0.549349] acpiphp: Slot [14] registered [ 0.550562] acpiphp: Slot [15] registered [ 0.551401] acpiphp: Slot [16] registered [ 0.552315] acpiphp: Slot [17] registered [ 0.553423] acpiphp: Slot [18] registered [ 0.554542] acpiphp: Slot [19] registered [ 0.555291] acpiphp: Slot [20] registered [ 0.556329] acpiphp: Slot [21] registered [ 0.557356] acpiphp: Slot [22] registered [ 0.558431] acpiphp: Slot [23] registered [ 0.559502] acpiphp: Slot [24] registered [ 0.560423] acpiphp: Slot [25] registered [ 0.561338] acpiphp: Slot [26] registered [ 0.562288] acpiphp: Slot [27] registered [ 0.563360] acpiphp: Slot [28] registered [ 0.565427] acpiphp: Slot [29] registered [ 0.566361] acpiphp: Slot [30] registered [ 0.567314] acpiphp: Slot [31] registered [ 0.568225] PCI host bridge to bus 0000:00 [ 0.569031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.570146] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.571025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.572020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.573051] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.574050] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.575442] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.579719] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.583048] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.589028] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.592079] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.593018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.594037] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.595024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.598733] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.599906] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.600047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.603495] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.605895] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.613020] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.616020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.643829] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.646020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.649020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.656064] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.684417] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.687031] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.690030] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.697030] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.722725] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.726025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.729021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.736054] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.768417] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.771044] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.774021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.781025] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.809767] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.812022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.815022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.822019] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.863921] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.866047] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.869041] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.877023] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.936420] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.945403] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.948379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.951317] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.953342] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.963754] iommu: Default domain type: Passthrough [ 0.964382] SCSI subsystem initialized [ 0.966341] ACPI: bus type USB registered [ 0.967480] usbcore: registered new interface driver usbfs [ 0.970289] usbcore: registered new interface driver hub [ 0.971240] usbcore: registered new device driver usb [ 0.973790] pps_core: LinuxPPS API ver. 1 registered [ 0.976013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.979127] PTP clock support registered [ 0.985212] EDAC MC: Ver: 3.0.0 [ 0.989313] PCI: Using ACPI for IRQ routing [ 0.990883] NetLabel: Initializing [ 0.993022] NetLabel: domain hash size = 128 [ 0.994010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.996385] NetLabel: unlabeled traffic allowed by default [ 0.998707] vgaarb: loaded [ 1.002083] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.004014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.013353] clocksource: Switched to clocksource kvm-clock [ 2.121271] VFS: Disk quotas dquot_6.6.0 [ 2.128426] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.140876] *** VALIDATE ramfs *** [ 2.142168] *** VALIDATE hugetlbfs *** [ 2.151829] pnp: PnP ACPI init [ 2.186918] pnp: PnP ACPI: found 6 devices [ 2.344822] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 2.352439] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 2.361545] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 2.368792] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 2.375821] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 2.383958] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 2.396478] NET: Registered protocol family 2 [ 2.402134] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 2.424478] tcp_listen_portaddr_hash hash table entries: 4096 (order: 6, 360448 bytes, vmalloc) [ 2.433219] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 2.455995] TCP bind hash table entries: 65536 (order: 10, 5242880 bytes, vmalloc) [ 2.489470] TCP: Hash tables configured (established 65536 bind 65536) [ 2.533559] MPTCP token hash table entries: 8192 (order: 7, 786432 bytes, vmalloc) [ 2.556972] UDP hash table entries: 4096 (order: 7, 786432 bytes, vmalloc) [ 2.566441] UDP-Lite hash table entries: 4096 (order: 7, 786432 bytes, vmalloc) [ 2.579603] NET: Registered protocol family 1 [ 2.598937] RPC: Registered named UNIX socket transport module. [ 2.607378] RPC: Registered udp transport module. [ 2.608918] RPC: Registered tcp transport module. [ 2.621424] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.630961] NET: Registered protocol family 44 [ 2.632450] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 2.639836] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 2.643403] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.650283] PCI: CLS 0 bytes, default 64 [ 2.654443] Unpacking initramfs... [ 10.786332] debug: unmapping init [mem 0xffff9cd93cbe3000-0xffff9cd93ffbffff] [ 10.802695] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 10.805044] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 10.807868] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 10.869039] cryptomgr_test (65) used greatest stack depth: 14248 bytes left [ 15.408798] Initialise system trusted keyrings [ 15.410346] Key type blacklist registered [ 15.412513] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 15.611841] zbud: loaded [ 15.665234] *** VALIDATE nfs *** [ 15.670843] *** VALIDATE nfs4 *** [ 15.676642] pstore: using deflate compression [ 15.709296] Platform Keyring initialized [ 15.720309] cryptomgr_test (73) used greatest stack depth: 14024 bytes left [ 15.755547] cryptomgr_test (84) used greatest stack depth: 13976 bytes left [ 15.812398] cryptomgr_test (86) used greatest stack depth: 13800 bytes left [ 15.915852] cryptomgr_test (94) used greatest stack depth: 13640 bytes left [ 16.364828] NET: Registered protocol family 38 [ 16.366419] Key type asymmetric registered [ 16.367862] Asymmetric key parser 'x509' registered [ 16.369699] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 16.372830] io scheduler mq-deadline registered [ 16.374434] io scheduler kyber registered [ 16.376466] io scheduler bfq registered [ 16.378414] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 16.384712] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 16.390585] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 16.398785] ACPI: Power Button [PWRF] [ 18.186280] hrtimer: interrupt took 12280694 ns [ 22.366949] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 28.905287] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 40.590045] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 47.390677] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 60.161854] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 60.296609] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 60.385143] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 60.409143] Non-volatile memory driver v1.3 [ 60.414651] Linux agpgart interface v0.103 [ 60.930196] virtio_blk virtio1: [vda] 131896 512-byte logical blocks (67.5 MB/64.4 MiB) [ 60.941628] vda: detected capacity change from 0 to 67530752 [ 61.027589] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 61.030297] vdb: detected capacity change from 0 to 1073741824 [ 61.140858] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 61.143712] vdc: detected capacity change from 0 to 2621440000 [ 61.247753] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 61.258509] vdd: detected capacity change from 0 to 2621440000 [ 61.328453] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 61.338638] vde: detected capacity change from 0 to 4294967296 [ 61.369713] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 61.417473] vdf: detected capacity change from 0 to 4294967296 [ 61.474562] libphy: Fixed MDIO Bus: probed [ 61.515737] usbcore: registered new interface driver usbserial_generic [ 61.521386] usbserial: USB Serial support registered for generic [ 61.532517] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 61.561056] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 61.571117] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 61.587964] mousedev: PS/2 mouse device common for all mice [ 61.609790] rtc_cmos 00:05: RTC can wake from S4 [ 61.612162] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 61.627862] rtc_cmos 00:05: registered as rtc0 [ 61.649522] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 61.663908] intel_pstate: CPU model not supported [ 61.673521] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 61.702391] hid: raw HID events driver (C) Jiri Kosina [ 61.706996] usbcore: registered new interface driver usbhid [ 61.707623] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 61.708659] usbhid: USB HID core driver [ 61.709023] drop_monitor: Initializing network drop monitor service [ 61.740913] Initializing XFRM netlink socket [ 61.779551] NET: Registered protocol family 10 [ 61.826398] Segment Routing with IPv6 [ 61.827948] NET: Registered protocol family 17 [ 61.835805] mpls_gso: MPLS GSO support [ 61.856184] RAS: Correctable Errors collector initialized. [ 61.863765] AVX version of gcm_enc/dec engaged. [ 61.865477] AES CTR mode by8 optimization enabled [ 62.094237] cryptomgr_test (159) used greatest stack depth: 13384 bytes left [ 62.249558] sched_clock: Marking stable (62249010092, 0)->(65588583497, -3339573405) [ 62.272664] registered taskstats version 1 [ 62.276925] Loading compiled-in X.509 certificates [ 62.281706] zswap: loaded using pool lzo/zbud [ 62.437416] Key type big_key registered [ 62.519935] Key type encrypted registered [ 62.521644] ima: No TPM chip found, activating TPM-bypass! [ 62.538321] ima: Allocated hash algorithm: sha1 [ 62.540051] ima: No architecture policies found [ 62.570785] evm: Initialising EVM extended attributes: [ 62.572810] evm: security.selinux [ 62.573816] evm: security.ima [ 62.575075] evm: security.capability [ 62.575910] evm: HMAC attrs: 0x1 [ 62.590246] rtc_cmos 00:05: setting system clock to 2025-04-01 07:34:14 UTC (1743492854) [ 62.705614] debug: unmapping init [mem 0xffffffff8c403000-0xffffffff8c5fffff] [ 62.715714] debug: unmapping init [mem 0xffffffff8aa71000-0xffffffff8af02fff] [ 62.722856] Write protecting the kernel read-only data: 30720k [ 62.732198] debug: unmapping init [mem 0xffffffff89003000-0xffffffff891fffff] [ 62.746422] debug: unmapping init [mem 0xffffffff8992f000-0xffffffff899fffff] [ 63.130427] systemd[1]: systemd 239 (239-82.el8_10.3) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 63.179979] systemd[1]: Detected virtualization kvm. [ 63.187281] systemd[1]: Detected architecture x86-64. [ 63.195813] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 63.333851] systemd[1]: No hostname configured. [ 63.342980] systemd[1]: Set hostname to . [ 63.353300] random: systemd: uninitialized urandom read (16 bytes read) [ 63.368606] systemd[1]: Initializing machine ID from random generator. [ 64.007035] random: ln: uninitialized urandom read (6 bytes read) [ 64.649626] random: systemd: uninitialized urandom read (16 bytes read) [ 64.652052] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 64.741164] random: systemd: uninitialized urandom read (16 bytes read) [ 64.762876] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 64.791752] random: systemd: uninitialized urandom read (16 bytes read) [ 64.810250] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ 68.203854] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 71.410834] device-mapper: uevent: version 1.0.3 [ 71.440610] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 72.149972] dracut-pre-udev (379) used greatest stack depth: 13312 bytes left [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 82.238973] virtio_net virtio0 ens2: renamed from eth0 [ 82.489936] random: fast init done [ 89.769789] scsi host0: ata_piix [ 89.991819] random: crng init done [ 89.993075] random: 5 urandom warning(s) missed due to ratelimiting [ 90.030486] scsi host1: ata_piix [ 90.053166] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 90.053176] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 91.613941] ip (528) used greatest stack depth: 13176 bytes left [ 91.897021] ip (529) used greatest stack depth: 11496 bytes left [ 99.800141] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 107.248936] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 119.686781] printk: systemd: 25 output lines suppressed due to ratelimiting [ 122.160348] SELinux: Disabled at runtime. [ 122.612510] systemd[1]: systemd 239 (239-82.el8_10.3) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 122.674466] systemd[1]: Detected virtualization kvm. [ 122.676481] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 128.732155] systemd[1]: initrd-switch-root.service: Succeeded. [ 128.764407] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 128.792170] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 128.833666] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 128.863888] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 128.931694] systemd[1]: Starting Journal Service... Starting Journal Service... [ 128.976711] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 129.201519] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ 129.330700] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ 129.423862] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 130.331310] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ OK ] Started Flush Journal to Persistent Storage. [ 136.376913] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 140.638173] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 141.136683] input: PC Speaker as /devices/platform/pcspkr/input/input5 [* ] A start job is running for Configur…only root support (16s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit) [ *** ] A start job is running for Configur…only root support (17s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit) [ ***] A start job is running for Configur…only root support (19s / no limit) [ **] A start job is running for Configur…only root support (19s / no limit) [ *] A start job is running for Configur…only root support (20s / no limit) [ **] A start job is running for Configur…only root support (21s / no limit) [ ***] A start job is running for Configur…only root support (21s / no limit) [ *** ] A start job is running for Configur…only root support (22s / no limit) [ *** ] A start job is running for Configur…only root support (23s / no limit)[ 152.142154] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [*** ] A start job is running for Configur…only root support (24s / no limit) [** ] A start job is running for Configur…only root support (24s / no limit) [* ] A start job is running for Configur…only root support (25s / no limit) [** ] A start job is running for Configur…only root support (26s / no limit) [*** ] A start job is running for Configur…only root support (26s / no limit) [ *** ] A start job is running for Configur…only root support (27s / no limit) [ *** ] A start job is running for Configur…only root support (28s / no limit)[ 156.427740] EDAC sbridge: Ver: 1.1.2 [ ***] A start job is running for Configur…only root support (28s / no limit) [ **] A start job is running for Configur…only root support (29s / no limit) [ *] A start job is running for Configur…only root support (30s / no limit) [ **] A start job is running for Configur…only root support (30s / no limit) [ ***] A start job is running for Configur…only root support (31s / no limit) [ *** ] A start job is running for Configur…only root support (32s / no limit) [ *** ] A start job is running for Configur…only root support (32s / no limit) [*** ] A start job is running for Configur…only root support (33s / no limit) [** ] A start job is running for Configur…only root support (33s / no limit) [* ] A start job is running for Configur…only root support (34s / no limit) [** ] A start job is running for Configur…only root support (34s / no limit) [*** ] A start job is running for Configur…only root support (35s / no limit) [ *** ] A start job is running for Configur…only root support (35s / no limit) [ *** ] A start job is running for Configur…only root support (36s / no limit) [ ***] A start job is running for Configur…only root support (36s / no limit) [ **] A start job is running for Configur…only root support (37s / no limit) [ *] A start job is running for Configur…only root support (37s / no limit) [ **] A start job is running for Configur…only root support (38s / no limit) [ ***] A start job is running for Configur…only root support (38s / no limit) [ *** ] A start job is running for Configur…only root support (39s / no limit) [ *** ] A start job is running for Configur…only root support (40s / no limit) [*** ] A start job is running for Configur…only root support (41s / no limit) [** ] A start job is running for Configur…only root support (41s / no limit)[ 169.884992] Key type dns_resolver registered [* ] A start job is running for Configur…only root support (42s / no limit) [** ] A start job is running for Configur…only root support (42s / no limit) [*** ] A start job is running for Configur…only root support (43s / no limit) [ *** ] A start job is running for Configur…only root support (43s / no limit)[ 171.982570] NFS: Registering the id_resolver key type [ 171.984375] Key type id_resolver registered [ 171.985652] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (44s / no limit) [ ***] A start job is running for Configur…only root support (44s / no limit)[ 173.200219] mount.nfs (977) used greatest stack depth: 10760 bytes left [ **] A start job is running for Configur…only root support (45s / no limit) [ *] A start job is running for Configur…only root support (45s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg644-server login: [ 296.081158] libcfs: loading out-of-tree module taints kernel. [ 296.111166] Key type ._llcrypt registered [ 296.130288] Key type .llcrypt registered [ 296.540218] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 336.210539] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 341.440986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 341.468306] alg: No test for adler32 (adler32-zlib) [ 343.623566] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 345.351831] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 345.360033] LNet: Accept secure, port 988 [ 347.439412] Key type lgssc registered [ 351.304733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 387.779772] ZFS: Loaded module v2.3.0-1, ZFS pool version 5000, ZFS filesystem version 5 [ 387.800137] modprobe (4345) used greatest stack depth: 5648 bytes left [ 393.973690] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 416.409582] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 432.665541] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 450.385492] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 493.297562] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 529.508552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 529.714610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 529.779907] ------------[ cut here ]------------ [ 529.781700] DEBUG_LOCKS_WARN_ON(!lockdep_enabled()) [ 529.781727] WARNING: CPU: 3 PID: 6532 at kernel/locking/lockdep.c:4713 lockdep_init_map_type+0x29d/0x410 [ 529.807889] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul ata_generic crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 529.879702] CPU: 3 PID: 6532 Comm: mount.lustre Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #7 [ 529.896528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 529.909404] RIP: 0010:lockdep_init_map_type+0x29d/0x410 [ 529.911059] Code: c0 0f 85 db fe ff ff 48 c7 c6 66 64 79 89 48 c7 c7 a7 4e 77 89 48 83 05 f0 df 25 03 01 e8 27 14 f5 ff 48 83 05 eb df 25 03 01 <0f> 0b 48 83 05 e9 df 25 03 01 48 83 05 e9 df 25 03 01 e9 a1 fe ff [ 529.954158] RSP: 0018:ffffbe930250b748 EFLAGS: 00010202 [ 529.959394] RAX: 0000000000000000 RBX: ffff9cd9be1cd010 RCX: 0000000000000001 [ 529.963839] RDX: 0000000000000001 RSI: 00000000ffff7fff RDI: ffff9cd9c1fde800 [ 529.978439] RBP: ffffffffc14b17c0 R08: 0000000000000000 R09: c0000000ffff7fff [ 529.990984] R10: 0000000000000001 R11: ffffbe930250b538 R12: 0000000000000002 [ 530.004330] R13: ffff9cd9afcd1000 R14: 0000000000000000 R15: 0000000000000001 [ 530.012481] FS: 00007f7085ff2b40(0000) GS:ffff9cd9c1e00000(0000) knlGS:0000000000000000 [ 530.023510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 530.047538] CR2: 0000563ed6d58038 CR3: 0000000124b7c003 CR4: 0000000000170ee0 [ 530.063382] Call Trace: [ 530.064702] ? show_regs.cold.9+0x22/0x2f [ 530.066359] ? __warn+0xc8/0x150 [ 530.067698] ? lockdep_init_map_type+0x29d/0x410 [ 530.074167] ? report_bug+0x113/0x140 [ 530.085668] ? do_error_trap+0xb6/0x130 [ 530.090751] ? do_invalid_op+0x46/0x60 [ 530.095790] ? lockdep_init_map_type+0x29d/0x410 [ 530.099827] ? invalid_op+0x14/0x20 [ 530.104521] ? lockdep_init_map_type+0x29d/0x410 [ 530.109718] ? lockdep_init_map_type+0x295/0x410 [ 530.115817] ldiskfs_enable_quotas+0x1b9/0x4a0 [ldiskfs] [ 530.128429] ldiskfs_fill_super+0x3a56/0x43c0 [ldiskfs] [ 530.138808] ? ldiskfs_calculate_overhead+0x670/0x670 [ldiskfs] [ 530.150817] ? mount_bdev+0x226/0x270 [ 530.159376] mount_bdev+0x226/0x270 [ 530.161690] ldiskfs_mount+0x19/0x30 [ldiskfs] [ 530.167281] legacy_get_tree+0x35/0x90 [ 530.169767] vfs_get_tree+0x2a/0x140 [ 530.171772] fc_mount+0x16/0x60 [ 530.174727] vfs_kern_mount+0x91/0x100 [ 530.180755] osd_mount+0x5c4/0x1080 [osd_ldiskfs] [ 530.186492] osd_device_init0+0x2e1/0xc20 [osd_ldiskfs] [ 530.189613] osd_device_alloc+0x22a/0x290 [osd_ldiskfs] [ 530.191676] obd_setup+0x196/0x430 [obdclass] [ 530.194457] class_setup+0x6f5/0x9f0 [obdclass] [ 530.196044] class_process_config+0x1658/0x2b60 [obdclass] [ 530.198636] do_lcfg+0x376/0x740 [obdclass] [ 530.200299] lustre_start_simple+0x8f/0x220 [obdclass] [ 530.203397] osd_start+0x6aa/0xb60 [ptlrpc] [ 530.206481] ? server_name2index+0x79/0xe0 [obdclass] [ 530.211803] ? lsi_prepare+0x2e7/0x690 [ptlrpc] [ 530.214160] server_fill_super+0x99/0x1190 [ptlrpc] [ 530.215980] ? obd_zombie_barrier+0x63/0x120 [obdclass] [ 530.218220] ? debug_mutex_init+0x43/0x60 [ 530.219630] lustre_fill_super+0x4a6/0x5e0 [lustre] [ 530.221598] ? lustre_mount+0x30/0x30 [lustre] [ 530.223639] mount_nodev+0x56/0xf0 [ 530.224915] lustre_mount+0x1c/0x30 [lustre] [ 530.241055] legacy_get_tree+0x35/0x90 [ 530.242490] vfs_get_tree+0x2a/0x140 [ 530.243762] do_mount+0xd84/0x1190 [ 530.245292] ksys_mount+0x11d/0x150 [ 530.246571] __x64_sys_mount+0x29/0x40 [ 530.247978] do_syscall_64+0xc1/0x450 [ 530.250397] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 530.264872] RIP: 0033:0x7f7082126dbe [ 530.276796] Code: 48 8b 0d cd 60 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 9a 60 39 00 f7 d8 64 89 01 48 [ 530.297889] RSP: 002b:00007ffdd5f54958 EFLAGS: 00000286 ORIG_RAX: 00000000000000a5 [ 530.303741] RAX: ffffffffffffffda RBX: 0000000000430cf6 RCX: 00007f7082126dbe [ 530.309772] RDX: 0000000000430cf6 RSI: 00007ffdd5f5b000 RDI: 0000000000b56940 [ 530.317202] RBP: 00007ffdd5f5b000 R08: 0000000000b56960 R09: 0000800100000000 [ 530.331408] R10: 0000000001000000 R11: 0000000000000286 R12: 0000000000000000 [ 530.337353] R13: 0000000000654920 R14: 00000000fffffff5 R15: 00000000fffffff5 [ 530.340452] ---[ end trace d3c86aaedb644998 ]--- [ 530.356242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.888592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 531.977199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 532.180819] Lustre: lustre-MDT0000: new disk, initializing [ 532.378305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.437249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 538.854579] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 559.974757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 560.169882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 560.416928] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 560.445194] Lustre: 7466:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 560.502449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 560.506317] Lustre: Skipped 1 previous similar message [ 560.700573] Lustre: lustre-MDT0001: new disk, initializing [ 560.854880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 560.920845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 560.929889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 567.444545] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 575.047164] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 590.805615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 591.016685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 591.524329] Lustre: 8424:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 591.579109] Lustre: lustre-OST0000: new disk, initializing [ 591.593240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 591.762315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 599.677879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 599.694245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 599.948530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 601.397952] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 622.422498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 622.577528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 622.809692] Lustre: 9442:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 622.882609] Lustre: lustre-OST0001: new disk, initializing [ 622.891944] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 623.058261] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 632.375776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 632.383271] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 632.576054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 634.091272] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 653.586574] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 667.487167] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 683.808299] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing check_logdir /tmp/testlogs/ [ 692.433294] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing yml_node [ 707.622605] Lustre: DEBUG MARKER: Client: 2.16.52.73 [ 713.163368] Lustre: DEBUG MARKER: MDS: 2.16.52.73 [ 719.692562] Lustre: DEBUG MARKER: OSS: 2.16.52.73 [ 723.859842] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 1 03:45:13 EDT 2025 [ 758.780053] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 762.597840] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 786.401976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 786.403179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.411305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 786.429211] Lustre: Skipped 2 previous similar messages [ 790.071738] Lustre: server umount lustre-MDT0000 complete [ 796.646555] LustreError: 6540:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743493588 with bad export cookie 2197636993387187096 [ 796.651421] LustreError: 6540:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 796.655395] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 796.685229] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.693932] LustreError: 9209:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 796.712387] Lustre: Skipped 1 previous similar message [ 796.723173] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 796.724394] LustreError: 9209:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 6 previous similar messages [ 796.757507] Lustre: Skipped 4 previous similar messages [ 801.767485] LustreError: 10377:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 801.768990] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 801.803779] Lustre: Skipped 1 previous similar message [ 801.804819] LustreError: 10377:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 803.620617] Lustre: server umount lustre-MDT0001 complete [ 811.240326] Lustre: server umount lustre-OST0000 complete [ 817.994595] Lustre: server umount lustre-OST0001 complete [ 829.923451] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 853.136277] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 877.213573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 892.326771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 905.923056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 920.158841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 933.468095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 933.628824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.230492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 934.319261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 934.479078] Lustre: lustre-MDT0000: new disk, initializing [ 934.666180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 934.760970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 942.245940] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 960.656041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 960.880865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 961.322674] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 961.387626] Lustre: 15719:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 961.518047] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 961.537723] Lustre: Skipped 1 previous similar message [ 961.642440] Lustre: lustre-MDT0001: new disk, initializing [ 961.915849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 962.009215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 962.015297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 969.556566] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 984.585429] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 992.762749] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1005.771117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1005.965693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1006.631891] Lustre: 16671:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1006.705730] Lustre: lustre-OST0000: new disk, initializing [ 1006.720506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1006.932668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1008.803181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1008.820636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1009.145050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1017.714940] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1032.218876] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1040.594195] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1041.169515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1048.665714] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1049.011891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1055.207455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1055.210688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.224895] LustreError: Skipped 1 previous similar message [ 1055.236025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1055.246768] Lustre: Skipped 2 previous similar messages [ 1055.293212] Lustre: Skipped 1 previous similar message [ 1058.845603] Lustre: server umount lustre-OST0000 complete [ 1069.551641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1069.554789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.560118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.610584] Lustre: server umount lustre-MDT0000 complete [ 1075.684219] LustreError: 15780:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.709866] LustreError: 15780:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 1079.521357] LustreError: 14947:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743493871 with bad export cookie 2197636993387188629 [ 1079.526195] LustreError: 14947:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1079.531539] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1080.078808] Lustre: server umount lustre-MDT0001 complete [ 1087.512479] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 03:51:16 (1743493876) [ 1100.171203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1101.222139] LustreError: 18802:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1101.437733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1106.917276] LustreError: 18802:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.728972] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1117.159628] LustreError: 18802:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.165632] LustreError: 18802:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 1124.848641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1125.073520] Lustre: 19500:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1131.975505] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1143.303239] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1151.952787] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1164.986680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1165.417478] Lustre: 20384:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1165.821674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1165.824588] Lustre: Skipped 1 previous similar message [ 1175.996851] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1188.233968] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1201.636470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1201.644561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.667854] Lustre: Skipped 1 previous similar message [ 1201.675732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1201.678298] Lustre: Skipped 2 previous similar messages [ 1203.367418] Lustre: server umount lustre-OST0000 complete [ 1212.385124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1212.397134] LustreError: Skipped 1 previous similar message [ 1212.407630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.435600] Lustre: Skipped 1 previous similar message [ 1216.600390] Lustre: server umount lustre-MDT0000 complete [ 1218.018866] LustreError: 18801:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.045654] LustreError: 18801:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 1222.897233] LustreError: 18781:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743494014 with bad export cookie 2197636993387189511 [ 1222.910259] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1222.917292] LustreError: 18781:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1223.692917] Lustre: server umount lustre-MDT0001 complete [ 1254.395195] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1262.307649] Key type lgssc unregistered [ 1262.976888] LNet: 21988:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.037594] LNet: Removed LNI 192.168.206.144@tcp [ 1266.625961] Key type .llcrypt unregistered [ 1266.627646] Key type ._llcrypt unregistered [ 1323.743640] Key type ._llcrypt registered [ 1323.745189] Key type .llcrypt registered [ 1324.030341] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1328.133803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.326351] alg: No test for adler32 (adler32-zlib) [ 1330.058232] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 1330.575778] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1330.580942] LNet: Accept secure, port 988 [ 1332.351270] Key type lgssc registered [ 1334.497751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1350.764188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1350.823430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.048374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.371466] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1362.401780] LustreError: 23381:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.420456] LustreError: 23381:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 1367.526289] LustreError: 23381:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.646178] LustreError: 23382:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.763981] LustreError: 23382:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1378.201421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1378.529877] Lustre: 24079:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1379.123110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1387.035411] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1400.308887] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1409.127435] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1422.843942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1423.320940] Lustre: 24965:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1423.336432] LustreError: 24972:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1423.724733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1433.078807] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1444.428550] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1447.448881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1472.994892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.996436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1473.008831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1478.115139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1478.142341] Lustre: Skipped 2 previous similar messages [ 1483.234136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1483.253613] Lustre: Skipped 1 previous similar message [ 1485.279398] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1485.746869] Lustre: server umount lustre-OST0000 complete [ 1493.472784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.488246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.495952] Lustre: Skipped 2 previous similar messages [ 1493.518462] Lustre: Skipped 1 previous similar message [ 1498.602419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.295148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1507.714634] Lustre: server umount lustre-MDT0000 complete [ 1508.833901] LustreError: 23382:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.855580] LustreError: 23382:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 1515.183057] LustreError: 23360:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743494307 with bad export cookie 11433454512574485908 [ 1515.212581] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1515.886471] Lustre: server umount lustre-MDT0001 complete [ 1548.402486] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1555.234615] Key type lgssc unregistered [ 1556.075601] LNet: 27012:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.158937] LNet: Removed LNI 192.168.206.144@tcp [ 1559.114479] Key type .llcrypt unregistered [ 1559.116889] Key type ._llcrypt unregistered [ 1608.394296] Key type ._llcrypt registered [ 1608.395861] Key type .llcrypt registered [ 1608.557095] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1612.383625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1612.433525] alg: No test for adler32 (adler32-zlib) [ 1613.997527] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 1614.695635] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1614.700434] LNet: Accept secure, port 988 [ 1616.663208] Key type lgssc registered [ 1619.127794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1634.129932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1634.191654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1636.414952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.813263] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1646.562811] LustreError: 28402:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.580971] LustreError: 28402:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 1651.681580] LustreError: 28402:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.801274] LustreError: 28402:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.993641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1658.104566] Lustre: 29100:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1658.555518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1665.048261] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1677.417524] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1686.195423] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1712.029372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1712.487261] Lustre: 30343:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1712.739914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1722.211059] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1731.236199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1733.134476] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1743.849994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1743.858373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.869634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.916959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1746.922987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.924191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.935412] Lustre: Skipped 1 previous similar message [ 1749.853955] Lustre: server umount lustre-OST0000 complete [ 1757.157987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.170093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.177858] Lustre: Skipped 1 previous similar message [ 1762.272320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.275090] Lustre: Skipped 1 previous similar message [ 1762.957743] Lustre: server umount lustre-MDT0000 complete [ 1767.396464] LustreError: 28402:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1767.433339] LustreError: 28402:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 1769.389971] LustreError: 28382:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743494561 with bad export cookie 18088826787118514784 [ 1769.410147] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1769.928923] Lustre: server umount lustre-MDT0001 complete [ 1797.423422] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1803.159977] Key type lgssc unregistered [ 1803.849917] LNet: 31947:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.901519] LNet: Removed LNI 192.168.206.144@tcp [ 1806.981084] Key type .llcrypt unregistered [ 1806.982499] Key type ._llcrypt unregistered [ 1860.416736] Key type ._llcrypt registered [ 1860.418370] Key type .llcrypt registered [ 1860.582945] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1865.512794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1865.646857] alg: No test for adler32 (adler32-zlib) [ 1867.321762] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 1867.920905] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1867.924697] LNet: Accept secure, port 988 [ 1869.697816] Key type lgssc registered [ 1871.871765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1885.748856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1885.782061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.587212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.672537] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1897.968357] LustreError: 33335:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1898.001490] LustreError: 33335:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 1903.080782] LustreError: 33335:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.193929] LustreError: 33335:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1911.768905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1912.158416] Lustre: 34036:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1912.292262] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1912.767247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1920.237462] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1933.252744] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1943.010716] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1957.703914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1958.344465] Lustre: 34922:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1958.795152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.508137] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1979.553159] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1982.306063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2002.917928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2002.933249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.955745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2002.997580] Lustre: Skipped 1 previous similar message [ 2007.947212] Lustre: server umount lustre-OST0000 complete [ 2014.440546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.447352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.455954] Lustre: Skipped 1 previous similar message [ 2019.808752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.811850] Lustre: Skipped 1 previous similar message [ 2020.853559] Lustre: server umount lustre-MDT0000 complete [ 2022.882295] LustreError: 33335:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.919683] LustreError: 33335:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 2027.638024] LustreError: 33316:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743494819 with bad export cookie 14260404036740385894 [ 2027.661702] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.366890] Lustre: server umount lustre-MDT0001 complete [ 2057.974169] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2064.764644] Key type lgssc unregistered [ 2065.422311] LNet: 36543:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.472351] LNet: Removed LNI 192.168.206.144@tcp [ 2068.506310] Key type .llcrypt unregistered [ 2068.508862] Key type ._llcrypt unregistered [ 2118.959138] Key type ._llcrypt registered [ 2118.962775] Key type .llcrypt registered [ 2119.186634] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2122.563291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2122.980136] alg: No test for adler32 (adler32-zlib) [ 2124.851591] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 2125.808112] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2125.834692] LNet: Accept secure, port 988 [ 2127.863269] Key type lgssc registered [ 2130.157207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2143.397808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2143.432380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.480761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.467481] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2156.009749] LustreError: 37931:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2156.033322] LustreError: 37931:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 2161.161358] LustreError: 37931:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.240961] LustreError: 37930:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.593644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2168.725376] Lustre: 38629:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2169.292557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2176.227886] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2187.176804] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2195.082918] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2206.683252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2206.918111] Lustre: 39514:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2207.200457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.428827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 2216.669091] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2227.641901] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2239.968289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2239.971332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.994698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2240.728944] Lustre: lustre-OST0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 2240.995928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.999136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2242.992431] Lustre: server umount lustre-OST0000 complete [ 2268.564988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.583766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2268.586062] Lustre: Skipped 2 previous similar messages [ 2271.588544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2271.591024] Lustre: Skipped 1 previous similar message [ 2275.043335] Lustre: server umount lustre-MDT0000 complete [ 2276.852245] LustreError: 37931:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.884384] LustreError: 37931:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 2281.152107] LustreError: 37912:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743495073 with bad export cookie 2340776138047012217 [ 2281.175521] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2287.889675] Lustre: server umount lustre-MDT0001 complete [ 2317.394963] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2324.124032] Key type lgssc unregistered [ 2324.801029] LNet: 41296:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.861387] LNet: Removed LNI 192.168.206.144@tcp [ 2327.725248] Key type .llcrypt unregistered [ 2327.728224] Key type ._llcrypt unregistered [ 2375.792037] Key type ._llcrypt registered [ 2375.797953] Key type .llcrypt registered [ 2376.032961] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2380.448550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2380.507289] alg: No test for adler32 (adler32-zlib) [ 2382.323793] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 2382.905589] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2382.909834] LNet: Accept secure, port 988 [ 2384.839231] Key type lgssc registered [ 2387.204742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2400.150940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2400.209125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.213107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2408.195965] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2412.514713] LustreError: 42684:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.550693] LustreError: 42684:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 2417.649951] LustreError: 42684:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.566325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2422.754717] LustreError: 42684:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.836615] Lustre: 43383:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2423.430618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2430.028982] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2440.069674] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2448.051894] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2460.814675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2461.358325] Lustre: 44267:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2461.677284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.323222] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2472.970224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2481.177841] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2492.145284] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 2493.408849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.416091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2493.419766] Lustre: Skipped 2 previous similar messages [ 2493.436629] Lustre: Skipped 2 previous similar messages [ 2497.205358] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 2498.030307] Lustre: server umount lustre-MDT0000 complete [ 2498.535627] LustreError: 42681:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.567089] LustreError: 42681:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 2503.659331] LustreError: 42684:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.685536] LustreError: 42684:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 2504.446374] LustreError: 44293:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743495296 with bad export cookie 13742780709339317977 [ 2504.452331] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2504.462736] LustreError: 44293:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2507.487725] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.44@tcp (stopping) [ 2508.773154] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.245148] Lustre: server umount lustre-MDT0001 complete [ 2530.437460] Lustre: server umount lustre-OST0000 complete [ 2567.828963] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2574.110710] Key type lgssc unregistered [ 2574.658770] LNet: 46194:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2575.718330] LNet: Removed LNI 192.168.206.144@tcp [ 2577.781811] Key type .llcrypt unregistered [ 2577.783444] Key type ._llcrypt unregistered [ 2626.224118] Key type ._llcrypt registered [ 2626.225698] Key type .llcrypt registered [ 2626.444456] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2629.869793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2630.076623] alg: No test for adler32 (adler32-zlib) [ 2631.633740] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 2632.320134] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2632.334956] LNet: Accept secure, port 988 [ 2634.315811] Key type lgssc registered [ 2636.242215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2650.871909] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2650.906483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2668.575467] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2674.719276] LustreError: 47557:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.144@tcp: failed processing log, type 4: rc = -110 [ 2704.607271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.363743] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2724.542308] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2892.940627] Lustre: server umount lustre-OST0000 complete [ 2931.813859] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2938.351766] Key type lgssc unregistered [ 2938.948243] LNet: 49124:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2940.016383] LNet: Removed LNI 192.168.206.144@tcp [ 2941.873296] Key type .llcrypt unregistered [ 2941.874798] Key type ._llcrypt unregistered [ 2991.436344] Key type ._llcrypt registered [ 2991.437812] Key type .llcrypt registered [ 2991.659078] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2995.453924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2995.982115] alg: No test for adler32 (adler32-zlib) [ 2997.798378] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 2998.428468] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2998.438862] LNet: Accept secure, port 988 [ 3000.335289] Key type lgssc registered [ 3002.669944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3016.662600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3016.721233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.805210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3025.509249] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3028.964241] LustreError: 50513:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3028.996830] LustreError: 50513:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 3034.093614] LustreError: 50513:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.212857] LustreError: 50512:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.344996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3040.501169] Lustre: 51212:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3040.871215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3048.035929] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3059.417058] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3067.156312] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3079.073601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3079.390719] Lustre: 52095:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3079.822144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3088.157745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 3089.820046] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3101.509353] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3107.810746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3107.829484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.851785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3108.840074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.849211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3108.849243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3112.995672] Lustre: server umount lustre-OST0000 complete [ 3122.656791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3122.671060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.688733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3122.692825] Lustre: Skipped 1 previous similar message [ 3126.187230] Lustre: server umount lustre-MDT0000 complete [ 3129.317405] LustreError: 50513:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3129.344665] LustreError: 50513:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 3132.502962] LustreError: 50494:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743495924 with bad export cookie 7107960220766752370 [ 3132.505978] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3132.557406] LustreError: 50494:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3133.131105] Lustre: server umount lustre-MDT0001 complete [ 3162.500201] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 3169.160921] Key type lgssc unregistered [ 3169.913899] LNet: 53698:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3170.989349] LNet: Removed LNI 192.168.206.144@tcp [ 3173.234839] Key type .llcrypt unregistered [ 3173.236584] Key type ._llcrypt unregistered [ 3224.050226] Key type ._llcrypt registered [ 3224.054644] Key type .llcrypt registered [ 3224.381351] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3227.596834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3227.944774] alg: No test for adler32 (adler32-zlib) [ 3230.200442] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 3230.935681] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3230.939275] LNet: Accept secure, port 988 [ 3232.943204] Key type lgssc registered [ 3234.974500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3248.201440] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3248.250706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3266.143584] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3272.287255] LustreError: 55059:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.144@tcp: failed processing log, type 4: rc = -110 [ 3302.111293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3311.383838] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3321.478542] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3333.117955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.964787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.080028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 3340.945487] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3342.312432] LustreError: 55977:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.328323] LustreError: 55977:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [ 3347.429565] LustreError: 55976:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.450809] LustreError: 55976:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 3352.549849] LustreError: 55977:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3352.590186] LustreError: 55977:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 3355.351367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3355.632855] Lustre: 56682:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3356.273086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3362.589197] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3372.325488] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3378.623722] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3385.312696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3385.315739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.348430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3386.853679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3386.857776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3386.867859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.883731] Lustre: Skipped 1 previous similar message [ 3388.670641] Lustre: server umount lustre-OST0000 complete [ 3421.665525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.680264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3425.517723] Lustre: server umount lustre-MDT0000 complete [ 3426.791287] LustreError: 55980:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3431.905160] LustreError: 55980:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3431.928989] LustreError: 55980:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 3433.040821] LustreError: 57701:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743496224 with bad export cookie 4173193181575719736 [ 3433.041608] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3433.074625] LustreError: 57701:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3439.995422] Lustre: server umount lustre-MDT0001 complete [ 3468.629783] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 3475.049462] Key type lgssc unregistered [ 3475.686453] LNet: 58453:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3476.774605] LNet: Removed LNI 192.168.206.144@tcp [ 3478.636794] Key type .llcrypt unregistered [ 3478.638399] Key type ._llcrypt unregistered [ 3529.466131] Key type ._llcrypt registered [ 3529.471027] Key type .llcrypt registered [ 3529.749077] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3533.942344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3534.056982] alg: No test for adler32 (adler32-zlib) [ 3535.437391] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 3535.810264] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3535.814858] LNet: Accept secure, port 988 [ 3537.591217] Key type lgssc registered [ 3539.867893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3552.933735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3552.988249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3554.710875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3560.688993] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3565.027661] LustreError: 59851:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3565.061184] LustreError: 59851:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 3570.148108] LustreError: 59851:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3574.644097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3574.891135] Lustre: 60548:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3575.333911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3581.601872] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3593.010736] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3601.530370] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3614.898722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3615.365339] Lustre: 61433:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3615.683827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3624.912579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 3626.755199] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3638.185847] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3660.771813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3660.777088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.814542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3660.824011] Lustre: Skipped 1 previous similar message [ 3665.359413] Lustre: server umount lustre-OST0000 complete [ 3672.558022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3672.562849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.605651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3672.610648] Lustre: Skipped 1 previous similar message [ 3676.130684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3676.137975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.638815] Lustre: server umount lustre-MDT0000 complete [ 3680.229386] LustreError: 61586:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.256654] LustreError: 61586:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 3685.357363] LustreError: 59851:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.377843] LustreError: 59851:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 3685.506311] LustreError: 59832:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743496477 with bad export cookie 18065885958121606075 [ 3685.512190] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3685.520402] LustreError: 59832:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3686.262471] Lustre: server umount lustre-MDT0001 complete [ 3716.569648] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 3723.633210] Key type lgssc unregistered [ 3724.272281] LNet: 63055:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3725.351394] LNet: Removed LNI 192.168.206.144@tcp [ 3727.137978] Key type .llcrypt unregistered [ 3727.139677] Key type ._llcrypt unregistered [ 3799.214459] Key type ._llcrypt registered [ 3799.215873] Key type .llcrypt registered [ 3799.478478] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3802.888995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3803.215681] alg: No test for adler32 (adler32-zlib) [ 3805.292626] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 3805.647357] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3805.654499] LNet: Accept secure, port 988 [ 3807.495301] Key type lgssc registered [ 3809.507460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3822.748391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3822.797969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3824.898750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3831.805427] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3834.342775] LustreError: 64738:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3834.354967] LustreError: 64738:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 3839.457860] LustreError: 64738:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3844.597210] LustreError: 64738:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.920940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3846.184848] Lustre: 65436:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3846.625135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3853.172905] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3864.140078] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3871.699620] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3883.674941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3883.912888] Lustre: 66322:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3884.271810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3894.196331] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3894.628182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 3905.057375] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3914.355502] Lustre: Failing over lustre-MDT0000 [ 3914.823697] Lustre: server umount lustre-MDT0000 complete [ 3915.233063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3915.239865] LustreError: 65932:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3915.248674] Lustre: Skipped 2 previous similar messages [ 3915.259048] LustreError: 65932:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 3920.359610] LustreError: 64739:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3920.390564] LustreError: 64739:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 3930.605443] LustreError: 64732:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3930.647658] LustreError: 64732:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 7 previous similar messages [ 3931.623223] Lustre: 64053:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743496707/real 1743496707] req@0000000059417d95 x1828188687728768/t0(0) o400->MGC192.168.206.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743496723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 3931.665778] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3931.807402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3942.175500] Lustre: *** cfs_fail_loc=135, val=0*** [ 3942.292789] LustreError: 67479:0:(obd_config.c:780:class_setup()) setup lustre-MDT0000 failed (-2) [ 3942.304429] LustreError: 67479:0:(obd_config.c:1995:class_config_llog_handler()) MGC192.168.206.144@tcp: cfg command failed: rc = -2 [ 3942.312587] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3942.312587] [ 3942.331869] LustreError: MGC192.168.206.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3942.357100] LustreError: 67466:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3942.360391] LustreError: 67466:0:(tgt_mount.c:2266:server_fill_super()) Unable to start targets: -2 [ 3942.370329] LustreError: 67466:0:(obd_config.c:833:class_cleanup()) Device 5 not setup [ 3942.421143] Lustre: server umount lustre-MDT0000 complete [ 3942.423085] LustreError: 67466:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3946.985476] LustreError: 64738:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.032085] LustreError: 64738:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 14 previous similar messages [ 3954.763538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3955.009490] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3955.026116] LustreError: 64730:0:(mgc_request.c:616:do_requeue()) failed processing log: -5 [ 3955.291373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3955.405067] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3958.477511] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3960.308675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.144@tcp (at 0@lo) [ 3960.345251] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3960.431341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 3961.933940] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3978.728954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3978.745457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3978.775860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.771663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.776786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.807488] Lustre: Skipped 1 previous similar message [ 3982.729546] Lustre: server umount lustre-OST0000 complete [ 3991.011993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.018398] Lustre: Skipped 1 previous similar message [ 3991.020509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3991.029263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3996.129909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3996.133595] Lustre: Skipped 2 previous similar messages [ 4001.254270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.256569] Lustre: Skipped 1 previous similar message [ 4004.321198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4004.759962] Lustre: server umount lustre-MDT0000 complete [ 4005.346181] LustreError: 64738:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4005.372577] LustreError: 64738:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 6 previous similar messages [ 4011.485629] LustreError: 64718:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743496803 with bad export cookie 9408927686685291289 [ 4011.502817] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4012.278618] Lustre: server umount lustre-MDT0001 complete [ 4041.862418] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 4048.665053] Key type lgssc unregistered [ 4049.424101] LNet: 69244:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4050.472233] LNet: Removed LNI 192.168.206.144@tcp [ 4052.200065] Key type .llcrypt unregistered [ 4052.201313] Key type ._llcrypt unregistered [ 4100.108328] Key type ._llcrypt registered [ 4100.109869] Key type .llcrypt registered [ 4100.395549] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4104.082259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4104.152115] alg: No test for adler32 (adler32-zlib) [ 4105.834360] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 4106.449161] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 4106.455365] LNet: Accept secure, port 988 [ 4108.255235] Key type lgssc registered [ 4110.668895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4125.038304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4125.085900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.058822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.049057] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4136.425465] LustreError: 70629:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4136.456566] LustreError: 70629:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 4141.551787] LustreError: 70629:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.673384] LustreError: 70629:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4149.303636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4149.656228] Lustre: 71330:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4150.304358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4157.144442] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4168.404270] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4176.790822] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4188.324784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4188.576681] Lustre: 72220:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4188.870591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4197.045355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 4197.262346] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4209.493867] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4219.981832] Lustre: Failing over lustre-MDT0000 [ 4220.539226] Lustre: server umount lustre-MDT0000 complete [ 4221.923795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4221.926391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.940617] LustreError: 71339:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4222.955075] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.071904] LustreError: 70629:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4228.096962] LustreError: 70629:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 6 previous similar messages [ 4235.262820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4235.389291] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4235.520151] Lustre: *** cfs_fail_loc=a05, val=0*** [ 4235.657331] LustreError: 73377:0:(obd_config.c:780:class_setup()) setup lustre-MDT0000 failed (-9) [ 4235.664549] LustreError: 73377:0:(obd_config.c:1995:class_config_llog_handler()) MGC192.168.206.144@tcp: cfg command failed: rc = -9 [ 4235.685485] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4235.685485] [ 4235.694776] LustreError: MGC192.168.206.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4235.703334] LustreError: 73366:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 4235.708588] LustreError: 73366:0:(tgt_mount.c:2266:server_fill_super()) Unable to start targets: -9 [ 4235.714371] LustreError: 73366:0:(obd_config.c:833:class_cleanup()) Device 5 not setup [ 4235.754485] Lustre: server umount lustre-MDT0000 complete [ 4235.757331] LustreError: 73366:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 4238.306069] LustreError: 71339:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4238.344862] LustreError: 71339:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 7 previous similar messages [ 4247.081994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4247.257103] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4247.271444] LustreError: 70621:0:(mgc_request.c:616:do_requeue()) failed processing log: -5 [ 4247.735459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4247.883432] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4249.794543] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4253.173506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.144@tcp (at 0@lo) [ 4253.208480] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4253.262439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 4254.655507] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4268.011176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4268.021420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.044195] Lustre: Skipped 1 previous similar message [ 4268.055425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4272.747658] Lustre: server umount lustre-OST0000 complete [ 4283.872901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4283.880213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.917242] Lustre: Skipped 2 previous similar messages [ 4283.927111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4283.929634] Lustre: Skipped 1 previous similar message [ 4286.397028] Lustre: server umount lustre-MDT0000 complete [ 4288.997513] LustreError: 71339:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4289.029963] LustreError: 71339:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 6 previous similar messages [ 4292.502266] LustreError: 70609:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743497084 with bad export cookie 13879445617218048344 [ 4292.505504] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4292.523371] LustreError: 70609:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4293.301138] Lustre: server umount lustre-MDT0001 complete [ 4319.885276] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 4326.028961] Key type lgssc unregistered [ 4326.604401] LNet: 75138:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4327.654217] LNet: Removed LNI 192.168.206.144@tcp [ 4329.429215] Key type .llcrypt unregistered [ 4329.430877] Key type ._llcrypt unregistered [ 4356.001160] Key type ._llcrypt registered [ 4356.002364] Key type .llcrypt registered [ 4356.176225] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 4356.702345] Key type .llcrypt unregistered [ 4356.703712] Key type ._llcrypt unregistered [ 4405.398162] Key type ._llcrypt registered [ 4405.399505] Key type .llcrypt registered [ 4405.627867] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4408.653906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4408.977836] alg: No test for adler32 (adler32-zlib) [ 4410.460499] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 4411.111370] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 4411.114641] LNet: Accept secure, port 988 [ 4412.967551] Key type lgssc registered [ 4414.891255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4428.016270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4428.047066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.899800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.023215] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4440.036705] LustreError: 76999:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4440.064711] LustreError: 76999:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 4445.155628] LustreError: 76999:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.125772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4450.281463] LustreError: 76999:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.467483] Lustre: 77698:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4451.030293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4456.286841] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4467.086653] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4474.192915] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4484.936496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4485.278203] Lustre: 78582:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4485.546823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.852809] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4499.917163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 4504.536986] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4519.537303] Lustre: server umount lustre-OST0000 complete [ 4525.539106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4525.547308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4525.567755] Lustre: Skipped 1 previous similar message [ 4525.591831] Lustre: Skipped 1 previous similar message [ 4530.673377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4530.677281] Lustre: Skipped 1 previous similar message [ 4535.779651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4535.787985] Lustre: Skipped 1 previous similar message [ 4535.959690] Lustre: server umount lustre-MDT0000 complete [ 4536.814241] Lustre: 76317:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743497312/real 1743497312] req@00000000eb33a02e x1828189322116224/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1743497328 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 4536.866864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4536.893444] LustreError: 78102:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4536.911056] LustreError: 78102:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 4537.823161] Lustre: 76314:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743497313/real 1743497313] req@00000000ba039d69 x1828189322116480/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1743497329 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 4541.928761] LustreError: 78954:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.964656] LustreError: 78954:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 4542.946123] Lustre: 76314:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743497317/real 1743497317] req@00000000a586321f x1828189322118016/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1743497333 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 4543.020427] LustreError: 76979:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743497334 with bad export cookie 6763931554084150188 [ 4543.044842] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4543.555411] Lustre: server umount lustre-MDT0001 complete [ 4570.075718] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 4575.841367] Key type lgssc unregistered [ 4576.361169] LNet: 80182:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4577.384213] LNet: Removed LNI 192.168.206.144@tcp [ 4579.119590] Key type .llcrypt unregistered [ 4579.122389] Key type ._llcrypt unregistered [ 4621.890650] Key type ._llcrypt registered [ 4621.894500] Key type .llcrypt registered [ 4622.062074] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4624.977748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4625.569224] alg: No test for adler32 (adler32-zlib) [ 4627.366899] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 4627.952909] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 4627.955711] LNet: Accept secure, port 988 [ 4629.793409] Key type lgssc registered [ 4631.927848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4645.149409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4645.199678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.179089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4653.233278] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4657.633234] LustreError: 81572:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.656846] LustreError: 81572:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 4662.754515] LustreError: 81572:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4666.597804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4666.879743] Lustre: 82270:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4667.200320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4673.643211] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4682.663818] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4689.583622] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4699.674730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4699.889165] Lustre: 83155:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4700.110475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4708.497338] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4717.581219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 4717.724366] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4726.753967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4726.762639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4726.780391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4727.783432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4727.786902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4727.787386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4732.901028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4732.913319] Lustre: Skipped 2 previous similar messages [ 4738.020393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4740.063149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4740.347531] Lustre: server umount lustre-OST0000 complete [ 4748.258718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4748.261462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4748.271975] Lustre: Skipped 1 previous similar message [ 4748.274235] Lustre: Skipped 1 previous similar message [ 4751.735919] Lustre: server umount lustre-MDT0000 complete [ 4753.402158] LustreError: 81567:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.411720] LustreError: 81567:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 4757.330267] LustreError: 82286:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743497549 with bad export cookie 7398819171398401572 [ 4757.332594] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4757.341606] LustreError: 82286:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4757.899555] Lustre: server umount lustre-MDT0001 complete [ 4782.407876] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 4788.223897] Key type lgssc unregistered [ 4788.846878] LNet: 84774:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4789.929396] LNet: Removed LNI 192.168.206.144@tcp [ 4791.814102] Key type .llcrypt unregistered [ 4791.815815] Key type ._llcrypt unregistered [ 4835.035065] Key type ._llcrypt registered [ 4835.040074] Key type .llcrypt registered [ 4835.230956] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4838.609639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4838.636031] alg: No test for adler32 (adler32-zlib) [ 4839.923438] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 4840.309838] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 4840.314505] LNet: Accept secure, port 988 [ 4842.143201] Key type lgssc registered [ 4843.846308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4855.503619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4855.555561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4857.429895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4863.103344] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4867.572656] LustreError: 86160:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4867.603572] LustreError: 86160:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 4872.681062] LustreError: 86160:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4876.785200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4876.999172] Lustre: 86856:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4877.396699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4883.013986] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4891.379732] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4898.538761] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4907.461978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4907.687814] Lustre: 87741:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4907.917153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4915.983458] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4925.180218] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4928.450951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 4947.936475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4947.939448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4947.982652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4948.960407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4948.968087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4948.969128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.997145] Lustre: Skipped 1 previous similar message [ 4954.091144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4954.093605] Lustre: Skipped 1 previous similar message [ 4959.203459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4959.211889] Lustre: Skipped 1 previous similar message [ 4960.224890] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4960.414262] Lustre: server umount lustre-OST0000 complete [ 4969.440347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4969.443458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.462325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4971.215970] Lustre: server umount lustre-MDT0000 complete [ 4974.560930] LustreError: 86159:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4974.578394] LustreError: 86159:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 4975.632657] LustreError: 87767:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743497767 with bad export cookie 9781002718518060630 [ 4975.636480] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4975.653739] LustreError: 87767:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4976.209933] Lustre: server umount lustre-MDT0001 complete [ 4998.742077] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 5003.972615] Key type lgssc unregistered [ 5004.531980] LNet: 89364:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5005.607340] LNet: Removed LNI 192.168.206.144@tcp [ 5007.225765] Key type .llcrypt unregistered [ 5007.227254] Key type ._llcrypt unregistered [ 5046.258783] Key type ._llcrypt registered [ 5046.259944] Key type .llcrypt registered [ 5046.386940] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5048.546538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5048.779407] alg: No test for adler32 (adler32-zlib) [ 5050.242772] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 5050.752310] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 5050.758702] LNet: Accept secure, port 988 [ 5052.599157] Key type lgssc registered [ 5054.290719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5064.705825] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5064.729639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5081.311582] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5087.455306] LustreError: 90720:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.144@tcp: failed processing log, type 4: rc = -110 [ 5118.367513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5125.808344] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5134.075584] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5141.983561] Lustre: server umount lustre-OST0000 complete [ 5157.079828] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 04:59:07 (1743497947) [ 5166.443665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5167.073210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5172.196916] LustreError: 92244:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.225608] LustreError: 92244:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 5172.341149] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5176.291781] LustreError: 92244:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5181.410858] LustreError: 92244:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5184.372535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5184.656051] Lustre: 92942:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5184.947379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5189.951582] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5198.515670] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5204.512587] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5213.221225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5213.373116] Lustre: 93824:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5213.569091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5221.192849] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5228.950466] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5237.114022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 5252.585836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5252.588449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5252.591229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5252.597523] Lustre: Skipped 1 previous similar message [ 5257.700350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5257.703153] Lustre: Skipped 2 previous similar messages [ 5258.216567] Lustre: server umount lustre-OST0000 complete [ 5266.915257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5266.918922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.924930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5267.941290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5269.698651] Lustre: server umount lustre-MDT0000 complete [ 5273.057889] LustreError: 92245:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5273.076705] LustreError: 92245:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 5274.572909] LustreError: 92224:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743498066 with bad export cookie 14116498536939743748 [ 5274.582296] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5275.034564] Lustre: server umount lustre-MDT0001 complete [ 5295.514073] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 5299.928277] Key type lgssc unregistered [ 5300.405910] LNet: 95590:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5301.484879] LNet: Removed LNI 192.168.206.144@tcp [ 5302.749871] Key type .llcrypt unregistered [ 5302.751641] Key type ._llcrypt unregistered [ 5339.137186] Key type ._llcrypt registered [ 5339.142362] Key type .llcrypt registered [ 5339.314741] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5341.812885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5341.838489] alg: No test for adler32 (adler32-zlib) [ 5343.126772] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 5343.496657] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 5343.500744] LNet: Accept secure, port 988 [ 5345.319195] Key type lgssc registered [ 5346.923136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5356.716067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5356.749938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.394086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.363687] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5368.802193] LustreError: 96978:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5368.827295] LustreError: 96978:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 5369.836870] LustreError: 96978:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5374.200149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5374.363471] Lustre: 97678:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5374.670433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5379.369064] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5387.407173] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5393.706565] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5402.138361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5402.305186] Lustre: 98562:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5402.484870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5404.546341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 5409.022361] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5416.628119] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5433.280934] Lustre: server umount lustre-OST0000 complete [ 5440.483971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5440.490196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5440.492179] Lustre: Skipped 1 previous similar message [ 5440.497391] Lustre: Skipped 1 previous similar message [ 5445.606208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5450.723447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5450.726678] Lustre: Skipped 2 previous similar messages [ 5451.743342] Lustre: 96296:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743498227/real 1743498227] req@00000000e224f840 x1828190300434816/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1743498243 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 5451.763945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.255595] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5452.382678] Lustre: server umount lustre-MDT0000 complete [ 5456.875355] Lustre: 96295:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743498232/real 1743498232] req@000000003bdf12d2 x1828190300437248/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1743498248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 5456.878793] LustreError: 98437:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5456.913173] LustreError: 98437:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 5456.934294] LustreError: 96958:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743498248 with bad export cookie 8260690048609742314 [ 5456.945525] LustreError: 96958:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5456.946920] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5457.296879] Lustre: server umount lustre-MDT0001 complete [ 5478.245988] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 5482.914923] Key type lgssc unregistered [ 5483.454739] LNet: 100180:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5484.518916] LNet: Removed LNI 192.168.206.144@tcp [ 5485.943360] Key type .llcrypt unregistered [ 5485.945071] Key type ._llcrypt unregistered [ 5513.949658] Key type ._llcrypt registered [ 5513.951046] Key type .llcrypt registered [ 5514.101376] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5516.490882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5516.530570] alg: No test for adler32 (adler32-zlib) [ 5517.919461] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 5518.363277] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 5518.368647] LNet: Accept secure, port 988 [ 5520.167218] Key type lgssc registered [ 5521.867382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5531.398810] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5531.433095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5549.151523] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5555.297522] LustreError: 101395:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.144@tcp: failed processing log, type 4: rc = -110 [ 5585.055270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5591.740095] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5599.162917] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5608.138409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5608.516019] LustreError: MGC192.168.206.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5608.529662] LustreError: 102301:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5608.534225] LustreError: 102301:0:(tgt_mount.c:2266:server_fill_super()) Unable to start targets: -2 [ 5608.542772] LustreError: 102301:0:(tgt_mount.c:1761:server_put_super()) no obd lustre-MDT0000 [ 5608.709471] Lustre: server umount lustre-MDT0000 complete [ 5608.711278] LustreError: 102301:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5629.919278] Lustre: 100739:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743498405/real 1743498405] req@000000002dd003f6 x1828190483909888/t0(0) o400->MGC192.168.206.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743498421 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 5629.932319] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5629.941537] LustreError: 101407:0:(mgc_request.c:616:do_requeue()) failed processing log: -5 [ 5650.565423] Lustre: server umount lustre-OST0000 complete [ 5661.381783] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 5677.852727] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5692.309921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5701.238071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5710.058676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5719.482560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5727.946565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5728.045190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5728.362783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5728.403872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5728.463251] Lustre: lustre-MDT0000: new disk, initializing [ 5728.603831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5728.637917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5733.270139] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5746.635739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5746.743594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5746.897054] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5746.927732] Lustre: 105844:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5746.966887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5746.969516] Lustre: Skipped 1 previous similar message [ 5747.023280] Lustre: lustre-MDT0001: new disk, initializing [ 5747.165814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5747.236346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5747.241286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5751.585145] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5761.469676] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5767.357505] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5776.159127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5776.281651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5776.555180] Lustre: 106795:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5776.589722] Lustre: lustre-OST0000: new disk, initializing [ 5776.594412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5776.706205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5782.612542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5782.625940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5782.800789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5783.304196] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5793.948398] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5799.403848] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5799.693888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5804.500831] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5804.852528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5808.616548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.626183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5808.630893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5808.638747] Lustre: Skipped 1 previous similar message [ 5808.642131] LustreError: Skipped 1 previous similar message [ 5813.857404] Lustre: server umount lustre-OST0000 complete [ 5823.969112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5823.972905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.986720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5824.002972] Lustre: Skipped 1 previous similar message [ 5824.008064] Lustre: Skipped 1 previous similar message [ 5825.337331] Lustre: server umount lustre-MDT0000 complete [ 5829.652203] LustreError: 105075:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743498621 with bad export cookie 10476236198265046826 [ 5829.655187] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5829.660136] LustreError: 105075:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5830.185182] Lustre: server umount lustre-MDT0001 complete [ 5843.509521] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 05:10:33 (1743498633) [ 5848.074406] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 5867.326448] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 5882.809118] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5899.857951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5911.032155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5919.184493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5927.775475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5935.424836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5935.529425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5935.873969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5935.928971] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5935.988329] Lustre: lustre-MDT0000: new disk, initializing [ 5936.136980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5936.158693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5940.452927] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5951.537574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5951.663067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5951.819447] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5951.845863] Lustre: 112304:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5951.881759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5951.884309] Lustre: Skipped 1 previous similar message [ 5951.957255] Lustre: lustre-MDT0001: new disk, initializing [ 5952.077081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5952.131444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5952.137460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5956.510718] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5965.241180] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5970.279155] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5978.043022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5978.161462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5978.444073] Lustre: 113253:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5978.481964] Lustre: lustre-OST0000: new disk, initializing [ 5978.490613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5978.586474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5984.888180] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5986.381969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5986.400045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5986.559358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5993.431577] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5998.334028] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5998.645482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6003.024606] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6003.295036] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6007.271228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6007.274126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6007.281544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6007.284274] Lustre: Skipped 1 previous similar message [ 6011.738868] Lustre: server umount lustre-OST0000 complete [ 6017.505929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6017.512250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6017.515914] Lustre: Skipped 2 previous similar messages [ 6017.520436] Lustre: Skipped 1 previous similar message [ 6022.625590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6022.628937] Lustre: Skipped 2 previous similar messages [ 6030.303136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6030.463660] Lustre: server umount lustre-MDT0000 complete [ 6032.865945] LustreError: 112693:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6032.881942] LustreError: 112693:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 4 previous similar messages [ 6034.859652] LustreError: 114660:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743498826 with bad export cookie 10476236198265047708 [ 6034.866323] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6034.871630] LustreError: 114660:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6035.226338] Lustre: server umount lustre-MDT0001 complete [ 6044.337662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6044.718129] LustreError: 115339:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6044.785357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6048.499965] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6050.274088] LustreError: 115339:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6054.375828] LustreError: 115339:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.839312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6057.935975] Lustre: 116036:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6062.392977] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6069.228766] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6074.235891] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6081.949472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6082.266161] Lustre: 116920:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6088.473593] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6094.875253] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6103.498837] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 6109.154262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6109.158739] LustreError: Skipped 1 previous similar message [ 6109.162137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6109.172186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6109.176624] Lustre: Skipped 2 previous similar messages [ 6109.456633] Lustre: server umount lustre-OST0000 complete [ 6114.785053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6123.603846] Lustre: server umount lustre-MDT0000 complete [ 6125.538421] LustreError: 115363:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6125.559911] LustreError: 115363:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 6127.278642] LustreError: 115320:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743498919 with bad export cookie 10476236198265048590 [ 6127.283241] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6127.283791] LustreError: 115320:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6127.585923] Lustre: server umount lustre-MDT0001 complete [ 6145.861182] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 6149.619467] Key type lgssc unregistered [ 6149.992551] LNet: 118620:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6151.017581] LNet: Removed LNI 192.168.206.144@tcp [ 6152.120758] Key type .llcrypt unregistered [ 6152.122230] Key type ._llcrypt unregistered [ 6171.744040] Key type ._llcrypt registered [ 6171.747075] Key type .llcrypt registered [ 6171.879451] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 6192.995323] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 6195.213553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6195.550921] alg: No test for adler32 (adler32-zlib) [ 6196.841349] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 6197.204164] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 6197.207880] LNet: Accept secure, port 988 [ 6198.879528] Key type lgssc registered [ 6200.242212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6211.185508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6219.445310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6226.880930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6235.899531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6244.244973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6244.308336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6244.332265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6245.679566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6245.736240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6245.791681] Lustre: lustre-MDT0000: new disk, initializing [ 6245.904422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6245.946567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6250.308989] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6261.837874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6261.949330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6262.083702] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6262.111235] Lustre: 122499:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6262.150538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6262.152734] Lustre: Skipped 1 previous similar message [ 6262.231801] Lustre: lustre-MDT0001: new disk, initializing [ 6262.342594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6262.390097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6262.398399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6267.046556] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6276.325958] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6281.754199] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6289.879630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6289.974429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6290.243934] Lustre: 123449:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6290.302925] Lustre: lustre-OST0000: new disk, initializing [ 6290.309940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6290.428521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6296.946989] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6299.206642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6299.213508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6299.365562] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6306.328267] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6311.341976] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6311.614835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6315.905905] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6316.174317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6320.098451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.106832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6320.110658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6320.120746] Lustre: Skipped 1 previous similar message [ 6324.667947] Lustre: server umount lustre-OST0000 complete [ 6330.336657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6330.342597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6330.348126] Lustre: Skipped 2 previous similar messages [ 6330.354274] Lustre: Skipped 1 previous similar message [ 6335.462911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6335.468977] Lustre: Skipped 1 previous similar message [ 6340.580136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6343.135179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6343.383960] Lustre: server umount lustre-MDT0000 complete [ 6347.628103] LustreError: 121729:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499139 with bad export cookie 13403545973753104111 [ 6347.641834] LustreError: 121729:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6347.644926] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6348.032325] Lustre: server umount lustre-MDT0001 complete [ 6360.705975] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:19:11 (1743499151) [ 6368.231377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6368.667866] LustreError: 125726:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6368.678291] LustreError: 125726:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 6368.750923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6372.526180] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6373.859710] LustreError: 125726:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6378.979759] LustreError: 125726:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6382.405795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6382.501456] Lustre: 126422:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6382.793463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6386.862247] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6393.408175] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6398.294155] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6403.553919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6403.554282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.567316] Lustre: Skipped 1 previous similar message [ 6403.570876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6403.574505] Lustre: Skipped 2 previous similar messages [ 6406.941678] Lustre: server umount lustre-MDT0000 complete [ 6407.652333] LustreError: 125720:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6407.667555] LustreError: 125720:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 6410.992498] LustreError: 125706:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499202 with bad export cookie 13403545973753104993 [ 6410.995446] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6411.002934] LustreError: 125706:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6411.297609] Lustre: server umount lustre-MDT0001 complete [ 6422.145704] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:20:12 (1743499212) [ 6429.406044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6446.049561] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6452.191590] LustreError: 127905:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.144@tcp: failed processing log, type 4: rc = -110 [ 6482.015337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6487.524449] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6493.403773] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6495.627973] Lustre: server umount lustre-OST0000 complete [ 6506.450890] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:21:37 (1743499297) [ 6513.561466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6513.982114] LustreError: 129235:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6514.074195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6517.949628] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6523.365771] LustreError: 129234:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6523.373167] LustreError: 129234:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 6526.784316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6526.898947] Lustre: 129931:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6530.729239] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6537.239076] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6541.673688] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6548.862272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6549.090589] Lustre: 130813:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6549.245979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6549.248578] Lustre: Skipped 1 previous similar message [ 6554.785451] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6560.808682] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6568.930539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6568.933203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6568.940283] Lustre: Skipped 1 previous similar message [ 6568.952808] Lustre: Skipped 1 previous similar message [ 6573.538751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6573.550226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6575.060591] Lustre: server umount lustre-MDT0000 complete [ 6578.642731] LustreError: 129217:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499370 with bad export cookie 13403545973753105658 [ 6578.649374] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6578.650537] LustreError: 129217:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6578.888811] Lustre: server umount lustre-MDT0001 complete [ 6592.728222] Lustre: server umount lustre-OST0000 complete [ 6602.916112] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:23:13 (1743499393) [ 6609.310651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.667604] LustreError: 132538:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6613.240571] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6620.941224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6621.073684] Lustre: 133235:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6621.327928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6621.334020] Lustre: Skipped 1 previous similar message [ 6625.087471] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6630.390560] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6634.799655] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6641.359423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6641.556649] Lustre: 134116:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6647.208825] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6653.175411] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6655.026370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 6657.278720] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6657.567960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6661.165587] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6661.445927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6665.186539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6665.186895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6665.187441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6665.187446] Lustre: Skipped 4 previous similar messages [ 6665.192166] Lustre: Skipped 1 previous similar message [ 6669.678959] Lustre: server umount lustre-OST0000 complete [ 6679.778623] Lustre: server umount lustre-MDT0000 complete [ 6680.545534] LustreError: 132531:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6680.553887] LustreError: 132531:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 6683.249856] LustreError: 133250:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499475 with bad export cookie 13403545973753106687 [ 6683.252878] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6683.257693] LustreError: 133250:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6683.534629] Lustre: server umount lustre-MDT0001 complete [ 6693.438691] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:24:44 (1743499484) [ 6700.086915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6717.727900] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6723.871569] LustreError: 136298:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.144@tcp: failed processing log, type 4: rc = -110 [ 6753.695542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6753.698574] Lustre: Skipped 1 previous similar message [ 6759.144835] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6764.935280] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6770.961813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6771.341652] LustreError: 136324:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6771.483185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 6775.044800] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6782.851443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6782.938843] Lustre: 137918:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6786.535365] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6792.111645] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6796.337966] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6800.366272] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6800.686395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6804.215936] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6804.441674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6808.033596] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6808.037951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6808.048215] Lustre: Skipped 3 previous similar messages [ 6808.057021] Lustre: Skipped 5 previous similar messages [ 6820.319098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6820.458479] Lustre: server umount lustre-MDT0000 complete [ 6823.729449] LustreError: 136304:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499615 with bad export cookie 13403545973753107709 [ 6823.736694] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6823.984562] Lustre: server umount lustre-MDT0001 complete [ 6838.123520] Lustre: server umount lustre-OST0000 complete [ 6849.318218] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:27:20 (1743499640) [ 6856.592319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6873.183780] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6879.331485] LustreError: 140094:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.144@tcp: failed processing log, type 4: rc = -110 [ 6913.065922] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6918.525667] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6925.433373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6925.791085] LustreError: 140121:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6925.801720] LustreError: 140121:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [ 6925.921040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6929.110274] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6936.582092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6936.679939] Lustre: 141717:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6940.503314] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6946.331533] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6950.763927] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6957.657398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6957.749057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6957.927554] Lustre: lustre-OST0001: new disk, initializing [ 6957.932507] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6962.833966] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6966.809976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6966.813177] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6966.940663] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6970.719300] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6975.067406] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6975.335630] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6979.081359] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6979.342916] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6982.124417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.126199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6982.147272] Lustre: Skipped 2 previous similar messages [ 6982.149967] Lustre: Skipped 6 previous similar messages [ 6982.157253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6987.286615] Lustre: server umount lustre-OST0000 complete [ 6992.354737] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6996.952904] Lustre: server umount lustre-OST0001 complete [ 7015.391144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7015.522746] Lustre: server umount lustre-MDT0000 complete [ 7018.941392] LustreError: 140103:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499810 with bad export cookie 13403545973753108654 [ 7018.947163] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7018.948977] LustreError: 140103:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7019.258391] Lustre: server umount lustre-MDT0001 complete [ 7024.188919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7028.289407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7032.017488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7035.847205] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7045.306845] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:30:35 (1743499835) [ 7047.727352] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 7049.956301] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 7052.233404] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:30:43 (1743499843) [ 7059.216313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7059.462604] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7059.519072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7059.684671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7059.688287] Lustre: Skipped 6 previous similar messages [ 7063.373719] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7071.544956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7071.633611] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7071.663623] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7071.704702] Lustre: 146741:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7071.710147] Lustre: 146741:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7075.689093] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7081.473539] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7085.924871] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7093.266539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7093.464421] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7099.262980] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7102.101373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 7105.574592] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7109.616486] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7109.872873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7113.983961] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7114.192516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7117.284070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7117.292896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7117.304070] Lustre: Skipped 4 previous similar messages [ 7117.311644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7117.314233] Lustre: Skipped 11 previous similar messages [ 7122.305565] Lustre: server umount lustre-OST0000 complete [ 7144.362687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7150.484981] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7155.174376] Lustre: lustre-MDT0001: haven't heard from client 3320abd3-8ced-4ac8-92c1-5f77237202f2 (at 192.168.206.44@tcp) in 31 seconds. I think it's dead, and I am evicting it. exp 0000000003f49fa4, cur 1743499947 expire 1743499917 last 1743499916 [ 7156.882316] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7157.988720] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7158.003273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7158.003971] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.206.144@tcp (at 0@lo) [ 7162.429829] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7162.759361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7166.769101] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7167.087302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7171.794656] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d0738fa1000.ost_server_uuid 50 [ 7174.010834] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d0738fa1000.ost_server_uuid in FULL state after 0 sec [ 7178.722864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7178.726248] LustreError: Skipped 1 previous similar message [ 7184.384169] Lustre: server umount lustre-OST0000 complete [ 7193.880629] Lustre: server umount lustre-MDT0000 complete [ 7194.087227] LustreError: 146043:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7194.094078] LustreError: 146043:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 8 previous similar messages [ 7197.443529] LustreError: 146023:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499989 with bad export cookie 13403545973753109830 [ 7197.446985] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7197.448614] LustreError: 146023:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7197.768750] Lustre: server umount lustre-MDT0001 complete [ 7213.938894] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7217.766053] Key type lgssc unregistered [ 7218.136418] LNet: 151483:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7219.173943] LNet: Removed LNI 192.168.206.144@tcp [ 7220.262052] Key type .llcrypt unregistered [ 7220.263806] Key type ._llcrypt unregistered [ 7251.210552] Key type ._llcrypt registered [ 7251.214773] Key type .llcrypt registered [ 7251.332294] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7253.195661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7253.244443] alg: No test for adler32 (adler32-zlib) [ 7254.425622] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7254.694543] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7254.697827] LNet: Accept secure, port 988 [ 7256.407204] Key type lgssc registered [ 7257.578749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7264.558746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7264.581323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7266.022718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7269.586722] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7275.489582] LustreError: 152868:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7275.500404] LustreError: 152868:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 7277.627546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7277.706357] Lustre: 153564:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7277.920639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7281.220972] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7287.385217] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7292.298241] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7299.072981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7299.236622] Lustre: 154450:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7299.392431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7305.024103] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7310.647056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7311.624653] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7317.710065] Lustre: Failing over lustre-MDT0000 [ 7317.914439] Lustre: server umount lustre-MDT0000 complete [ 7318.731572] LustreError: 152861:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7319.009802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7319.014294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7321.060964] LustreError: 152867:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7321.061883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7321.078103] LustreError: 152867:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 7321.083204] Lustre: Skipped 1 previous similar message [ 7326.178385] LustreError: 152867:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.185101] LustreError: 152867:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 7330.115512] LustreError: 152846:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.44@tcp arrived at 1743500122 with bad export cookie 9525663622909328340 [ 7330.125617] LustreError: 152846:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7331.300264] LustreError: 152867:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7331.308582] LustreError: 152867:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 7336.416784] Lustre: 152185:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743500112/real 1743500112] req@0000000059702f0c x1828192304259712/t0(0) o400->MGC192.168.206.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743500128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 7336.433116] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7337.090446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7341.538755] LustreError: 152862:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7341.554856] LustreError: 152862:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [ 7346.656920] LustreError: 152181:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d5b4d6d8 x1828192304264448/t0(0) o250->MGC192.168.206.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 7346.864923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7346.914531] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7350.544122] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7352.301045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.144@tcp (at 0@lo) [ 7352.305707] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7352.309668] Lustre: lustre-OST0000: haven't heard from client de96da4d-554a-4f82-ab08-5435fd1139a1 (at 192.168.206.44@tcp) in 34 seconds. I think it's dead, and I am evicting it. exp 00000000e193c364, cur 1743500144 expire 1743500114 last 1743500110 [ 7352.826648] Lustre: lustre-MDT0000: Denying connection for new client 054e5cbf-e383-41c0-84c2-a590f44f51b7 (at 192.168.206.44@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 7358.135579] Lustre: lustre-MDT0000: Denying connection for new client 054e5cbf-e383-41c0-84c2-a590f44f51b7 (at 192.168.206.44@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 7358.148984] Lustre: Skipped 1 previous similar message [ 7362.535083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.540652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7362.549140] Lustre: Skipped 1 previous similar message [ 7367.552717] Lustre: server umount lustre-OST0000 complete [ 7371.125177] LustreError: 156225:0:(ldlm_lib.c:2907:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 7371.129918] Lustre: 155597:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 7371.135619] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 7371.150207] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 7371.156246] LustreError: 155597:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003aa1d8eb x1828192304278784/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 7371.164964] LustreError: 155597:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7371.171575] LustreError: 155597:0:(fid_request.c:315:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 7371.178482] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 7371.398231] Lustre: server umount lustre-MDT0000 complete [ 7372.769371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.782046] Lustre: Skipped 1 previous similar message [ 7372.787636] LustreError: 152864:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7372.798670] LustreError: 152864:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 7375.378248] LustreError: 153581:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500167 with bad export cookie 9525663622909328438 [ 7375.388097] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7375.391923] LustreError: 153581:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7375.403485] LustreError: 152181:0:(import.c:699:ptlrpc_connect_import_locked()) can't connect to a closed import [ 7377.777534] Lustre: server umount lustre-MDT0001 complete [ 7394.607932] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7398.408392] Key type lgssc unregistered [ 7398.819549] LNet: 156974:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7399.845736] LNet: Removed LNI 192.168.206.144@tcp [ 7400.895703] Key type .llcrypt unregistered [ 7400.897185] Key type ._llcrypt unregistered [ 7430.727035] Key type ._llcrypt registered [ 7430.728576] Key type .llcrypt registered [ 7430.826922] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7432.538490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7432.618025] alg: No test for adler32 (adler32-zlib) [ 7433.771133] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7434.011562] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7434.015878] LNet: Accept secure, port 988 [ 7435.707059] Key type lgssc registered [ 7436.833124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7444.606502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7444.632993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7446.059328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7449.577916] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7455.204151] LustreError: 158357:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7455.214696] LustreError: 158357:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 7457.776621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7457.866053] Lustre: 159052:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7458.126401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7461.691330] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7467.085174] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7471.767451] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7478.046939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7478.223564] Lustre: 159937:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7478.385896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7483.255178] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7488.956891] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7490.679380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7496.162690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7496.170186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7496.176170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7496.178366] Lustre: Skipped 1 previous similar message [ 7496.183325] Lustre: Skipped 1 previous similar message [ 7500.777727] Lustre: server umount lustre-OST0000 complete [ 7504.353092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7504.356533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7504.362875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7506.402408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7506.403176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.570904] Lustre: server umount lustre-MDT0000 complete [ 7511.522313] LustreError: 158357:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7511.530657] LustreError: 158357:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 7513.704605] LustreError: 160498:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500305 with bad export cookie 356918654710302811 [ 7513.710863] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7513.711281] LustreError: 160498:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7514.021676] Lustre: server umount lustre-MDT0001 complete [ 7530.697427] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7534.409846] Key type lgssc unregistered [ 7534.850683] LNet: 161539:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7535.911863] LNet: Removed LNI 192.168.206.144@tcp [ 7536.983961] Key type .llcrypt unregistered [ 7536.985526] Key type ._llcrypt unregistered [ 7578.602222] Key type ._llcrypt registered [ 7578.607190] Key type .llcrypt registered [ 7578.733851] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7580.807982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7580.845490] alg: No test for adler32 (adler32-zlib) [ 7582.075118] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7582.385982] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7582.395040] LNet: Accept secure, port 988 [ 7584.087221] Key type lgssc registered [ 7585.325690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7593.363829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7593.396283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7594.792147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7598.094163] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7604.198696] LustreError: 163213:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7604.204952] LustreError: 163213:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 7604.914845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7605.002414] Lustre: 163910:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7605.204330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7608.538980] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7614.338505] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7618.562560] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7624.863072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7625.019490] Lustre: 164793:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7625.166576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7629.804075] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7634.815545] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7639.537097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 7645.665207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7645.668273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7645.676297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7649.763247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7649.765453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7649.767597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.776997] Lustre: Skipped 1 previous similar message [ 7654.886249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7654.888686] Lustre: Skipped 1 previous similar message [ 7658.975133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7659.146146] Lustre: server umount lustre-OST0000 complete [ 7665.121613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.126982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7668.800604] Lustre: server umount lustre-MDT0000 complete [ 7670.241356] LustreError: 163212:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7670.252380] LustreError: 163212:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 7672.508137] LustreError: 163926:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500464 with bad export cookie 7157014398743360553 [ 7672.509728] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7672.513799] LustreError: 163926:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7672.803500] Lustre: server umount lustre-MDT0001 complete [ 7688.758790] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7692.523530] Key type lgssc unregistered [ 7692.888527] LNet: 166388:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7693.927713] LNet: Removed LNI 192.168.206.144@tcp [ 7694.987772] Key type .llcrypt unregistered [ 7694.989573] Key type ._llcrypt unregistered [ 7723.523942] Key type ._llcrypt registered [ 7723.525520] Key type .llcrypt registered [ 7723.638678] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7725.665741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7725.701595] alg: No test for adler32 (adler32-zlib) [ 7727.034895] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7727.379972] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7727.384605] LNet: Accept secure, port 988 [ 7729.063189] Key type lgssc registered [ 7730.114543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7739.121048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7739.149926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.465475] Lustre: *** cfs_fail_loc=135, val=0*** [ 7740.511735] LustreError: 167883:0:(obd_config.c:780:class_setup()) setup lustre-MDT0000 failed (-2) [ 7740.515318] LustreError: 167883:0:(obd_config.c:1995:class_config_llog_handler()) MGC192.168.206.144@tcp: cfg command failed: rc = -2 [ 7740.519287] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7740.519287] [ 7740.524596] LustreError: MGC192.168.206.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7740.529021] LustreError: 167844:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 7740.531532] LustreError: 167844:0:(tgt_mount.c:2266:server_fill_super()) Unable to start targets: -2 [ 7740.539063] LustreError: 167844:0:(obd_config.c:833:class_cleanup()) Device 5 not setup [ 7740.642179] Lustre: server umount lustre-MDT0000 complete [ 7740.644240] LustreError: 167844:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7745.995140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7746.416049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7749.396396] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7756.769928] LustreError: 168273:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7756.784394] LustreError: 168273:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 7756.921944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7757.008847] Lustre: 168951:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7757.230739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7760.542945] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7765.895443] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7770.244748] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7772.640348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7772.643106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7772.647987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7776.736395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7776.737487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.251145] Lustre: server umount lustre-MDT0000 complete [ 7781.043109] LustreError: 169587:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500572 with bad export cookie 14306696105009610385 [ 7781.054517] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7781.289396] Lustre: server umount lustre-MDT0001 complete [ 7794.594471] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7797.984712] Key type lgssc unregistered [ 7798.307799] LNet: 170317:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7799.337522] LNet: Removed LNI 192.168.206.144@tcp [ 7800.296161] Key type .llcrypt unregistered [ 7800.297800] Key type ._llcrypt unregistered [ 7823.926867] Key type ._llcrypt registered [ 7823.928172] Key type .llcrypt registered [ 7824.035977] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7824.432903] Key type .llcrypt unregistered [ 7824.434367] Key type ._llcrypt unregistered [ 7845.516453] Key type ._llcrypt registered [ 7845.517815] Key type .llcrypt registered [ 7845.637502] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7847.247618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7847.274395] alg: No test for adler32 (adler32-zlib) [ 7848.448937] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7848.756940] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7848.760684] LNet: Accept secure, port 988 [ 7850.519394] Key type lgssc registered [ 7851.571377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7858.672312] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7858.695990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7875.295597] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7881.439271] LustreError: 172394:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.144@tcp: failed processing log, type 4: rc = -110 [ 7911.200238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7916.120301] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7921.745825] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7928.356330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.486588] LustreError: 171736:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ba8dc4b1 x1828192927090944/t0(0) o250->MGC192.168.206.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 7928.809326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7928.870898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 7932.238824] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7933.927345] LustreError: 172420:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.938244] LustreError: 172420:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 7939.049995] LustreError: 172420:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7939.085749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7939.179780] Lustre: 174016:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7939.411192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7942.642513] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7948.306234] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7952.549882] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7955.017311] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 7966.177388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7966.186320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7966.190075] Lustre: Skipped 1 previous similar message [ 7971.300359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7971.303336] Lustre: Skipped 1 previous similar message [ 7972.053553] Lustre: server umount lustre-OST0000 complete [ 7975.396053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7975.399984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7975.407060] Lustre: Skipped 1 previous similar message [ 7975.409661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7976.419605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7981.454713] Lustre: server umount lustre-MDT0000 complete [ 7981.536984] LustreError: 173315:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7981.545376] LustreError: 173315:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 7984.684032] LustreError: 172402:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500776 with bad export cookie 4102414705874189759 [ 7984.687815] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7984.690932] LustreError: 172402:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7984.969147] Lustre: server umount lustre-MDT0001 complete [ 7999.560696] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8002.950789] Key type lgssc unregistered [ 8003.311279] LNet: 176049:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8004.327362] LNet: Removed LNI 192.168.206.144@tcp [ 8005.218130] Key type .llcrypt unregistered [ 8005.219356] Key type ._llcrypt unregistered [ 8033.539884] Key type ._llcrypt registered [ 8033.541372] Key type .llcrypt registered [ 8033.637107] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8035.252881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8035.304129] alg: No test for adler32 (adler32-zlib) [ 8036.500270] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8036.747380] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8036.750469] LNet: Accept secure, port 988 [ 8038.415224] Key type lgssc registered [ 8039.472541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8046.624332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8046.647814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8048.095733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8051.478298] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8057.315946] LustreError: 177433:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8057.326355] LustreError: 177433:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 8059.341268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8059.424165] Lustre: 178131:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8059.641929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8063.118531] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8069.021473] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8073.466731] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8079.730139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8079.883234] Lustre: 179014:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8080.025811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8084.903893] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8090.232230] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8092.339199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 8099.359860] Lustre: Failing over lustre-MDT0000 [ 8099.510585] Lustre: server umount lustre-MDT0000 complete [ 8100.833880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8100.843273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.852924] LustreError: 178484:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8102.881971] LustreError: 178484:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8102.886053] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8102.892261] LustreError: 178484:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 8102.901979] Lustre: Skipped 1 previous similar message [ 8106.680388] LustreError: 179041:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.688393] LustreError: 179041:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 8111.800056] LustreError: 179577:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.815421] LustreError: 179577:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 8118.024612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8118.141595] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8118.244163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 8118.247945] Lustre: Skipped 2 previous similar messages [ 8118.324899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8118.377932] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8121.882131] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8121.973712] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8123.378016] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.144@tcp (at 0@lo) [ 8123.395608] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 8123.422097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 8126.063827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire in log lustre-MDT0000 [ 8127.957250] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight in log lustre-client [ 8141.793321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8141.800236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8141.807297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8141.810783] Lustre: Skipped 1 previous similar message [ 8142.276791] Lustre: server umount lustre-OST0000 complete [ 8148.962173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8148.962505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8148.970243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8148.970902] LustreError: Skipped 1 previous similar message [ 8148.976183] Lustre: Skipped 2 previous similar messages [ 8154.081581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8154.086244] Lustre: Skipped 1 previous similar message [ 8158.177151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8158.179866] Lustre: Skipped 1 previous similar message [ 8160.223429] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8160.350862] Lustre: server umount lustre-MDT0000 complete [ 8163.296815] LustreError: 178140:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8163.310188] LustreError: 178140:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 6 previous similar messages [ 8163.755577] LustreError: 177412:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500955 with bad export cookie 6555164129437238827 [ 8163.761700] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8163.764052] LustreError: 177412:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8164.054605] Lustre: server umount lustre-MDT0001 complete [ 8180.769545] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8184.234468] Key type lgssc unregistered [ 8184.611447] LNet: 181814:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8185.638943] LNet: Removed LNI 192.168.206.144@tcp [ 8186.837835] Key type .llcrypt unregistered [ 8186.839613] Key type ._llcrypt unregistered [ 8217.122994] Key type ._llcrypt registered [ 8217.124973] Key type .llcrypt registered [ 8217.260164] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8219.233577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8219.271187] alg: No test for adler32 (adler32-zlib) [ 8220.466252] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8220.771911] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8220.775259] LNet: Accept secure, port 988 [ 8222.439214] Key type lgssc registered [ 8223.547439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8231.198941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8231.216730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8232.648321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8235.718383] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8238.049291] LustreError: 183198:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8238.059426] LustreError: 183198:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 8242.152537] LustreError: 183198:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8243.173851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8243.249580] Lustre: 183895:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8243.434824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8246.474405] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8252.138441] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8256.804974] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8263.173048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8263.328366] Lustre: 184779:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8263.524233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8268.359245] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8273.675347] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8277.874158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 8281.196385] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 8283.535134] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 8293.859856] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 8308.707987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8308.708413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8308.709478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8308.717603] Lustre: Skipped 1 previous similar message [ 8310.926443] Lustre: server umount lustre-OST0000 complete [ 8315.361299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8315.366543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.372990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8315.374657] Lustre: Skipped 1 previous similar message [ 8318.946643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8318.948318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8320.484985] Lustre: server umount lustre-MDT0000 complete [ 8324.066246] LustreError: 184370:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8324.075721] LustreError: 184370:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 8324.080697] LustreError: 183178:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501115 with bad export cookie 7601274452667833191 [ 8324.082622] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8324.085072] LustreError: 183178:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8324.360873] Lustre: server umount lustre-MDT0001 complete [ 8341.062198] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8344.450788] Key type lgssc unregistered [ 8344.793794] LNet: 186542:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8345.830564] LNet: Removed LNI 192.168.206.144@tcp [ 8346.760903] Key type .llcrypt unregistered [ 8346.762515] Key type ._llcrypt unregistered [ 8376.570524] Key type ._llcrypt registered [ 8376.572022] Key type .llcrypt registered [ 8376.664896] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8378.458366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8378.477483] alg: No test for adler32 (adler32-zlib) [ 8379.614279] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8379.803929] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8379.806647] LNet: Accept secure, port 988 [ 8381.463239] Key type lgssc registered [ 8382.430685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8388.487021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8388.507312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8389.847965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8392.751408] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8399.330504] LustreError: 187925:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8399.337692] LustreError: 187925:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 8399.496849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8399.595891] Lustre: 188621:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8399.817653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8403.020638] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8408.217609] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8412.300417] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8417.963046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8418.093902] Lustre: 189503:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8418.241340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8422.237673] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8427.568223] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8434.669474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 8437.698898] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 8448.445244] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 8459.838877] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 8463.996603] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 8475.619632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8475.621330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8475.621496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8475.626317] Lustre: Skipped 1 previous similar message [ 8475.633627] Lustre: Skipped 1 previous similar message [ 8479.888481] Lustre: server umount lustre-OST0000 complete [ 8485.856528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8485.862119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8485.864776] Lustre: Skipped 1 previous similar message [ 8490.977139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8490.979921] Lustre: Skipped 2 previous similar messages [ 8496.096943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8496.102028] Lustre: Skipped 1 previous similar message [ 8497.119202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8497.224091] Lustre: server umount lustre-MDT0000 complete [ 8500.211597] LustreError: 190701:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501292 with bad export cookie 16979178793770462785 [ 8500.215234] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8500.217733] LustreError: 190701:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8500.482735] Lustre: server umount lustre-MDT0001 complete [ 8516.290119] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8519.264800] Key type lgssc unregistered [ 8519.600596] LNet: 192737:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8520.613529] LNet: Removed LNI 192.168.206.144@tcp [ 8521.509307] Key type .llcrypt unregistered [ 8521.510749] Key type ._llcrypt unregistered [ 8548.997166] Key type ._llcrypt registered [ 8548.998695] Key type .llcrypt registered [ 8549.097445] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8551.105367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8551.159151] alg: No test for adler32 (adler32-zlib) [ 8552.266427] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8552.488957] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8552.490759] LNet: Accept secure, port 988 [ 8554.135209] Key type lgssc registered [ 8555.063169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8560.940302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8560.962378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8562.341960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8565.269230] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8571.888709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8571.976530] Lustre: 194817:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8572.174755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8574.840480] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8579.697127] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8583.403748] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8588.922567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8589.079510] Lustre: 195700:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8589.247640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8593.944801] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8599.422617] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8608.692570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 8628.192258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8628.195038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8628.200453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.217212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8629.220190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.223064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8630.742311] Lustre: server umount lustre-OST0000 complete [ 8634.340080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.342058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8634.347926] Lustre: Skipped 1 previous similar message [ 8634.350182] Lustre: Skipped 1 previous similar message [ 8639.458023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.459877] Lustre: Skipped 1 previous similar message [ 8644.577103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8644.580109] Lustre: Skipped 1 previous similar message [ 8648.159142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8648.277705] Lustre: server umount lustre-MDT0000 complete [ 8649.697234] LustreError: 194826:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8649.710284] LustreError: 194826:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 8651.321145] LustreError: 194101:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501443 with bad export cookie 14682777941554196881 [ 8651.325408] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8651.325824] LustreError: 194101:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8651.585371] Lustre: server umount lustre-MDT0001 complete [ 8666.898899] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8669.911651] Key type lgssc unregistered [ 8670.250722] LNet: 197470:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8671.269497] LNet: Removed LNI 192.168.206.144@tcp [ 8672.094875] Key type .llcrypt unregistered [ 8672.095880] Key type ._llcrypt unregistered [ 8692.848101] Key type ._llcrypt registered [ 8692.849631] Key type .llcrypt registered [ 8692.962793] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8694.477202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8694.528496] alg: No test for adler32 (adler32-zlib) [ 8695.711522] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8695.919855] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8695.921697] LNet: Accept secure, port 988 [ 8697.575260] Key type lgssc registered [ 8698.604383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8704.825940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8704.839978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8706.183846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8708.826064] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8714.675890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8714.738663] Lustre: 199404:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8714.921902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8717.378294] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8721.661798] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8725.110485] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8730.199789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8730.346744] Lustre: 200287:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8730.546620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8731.565073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 8734.314186] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8738.385237] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8751.074841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8751.077353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8751.079071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8751.083412] Lustre: Skipped 1 previous similar message [ 8751.087905] Lustre: Skipped 1 previous similar message [ 8756.196219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8756.198622] Lustre: Skipped 1 previous similar message [ 8756.934019] Lustre: server umount lustre-OST0000 complete [ 8761.314245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8761.314621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8761.323983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8761.326024] Lustre: Skipped 1 previous similar message [ 8765.622933] Lustre: server umount lustre-MDT0000 complete [ 8766.433049] LustreError: 199413:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8766.439685] LustreError: 199413:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 8768.090364] LustreError: 198688:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501559 with bad export cookie 11354433392002741787 [ 8768.095733] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8768.100237] LustreError: 198688:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8768.327391] Lustre: server umount lustre-MDT0001 complete [ 8782.257882] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8785.121955] Key type lgssc unregistered [ 8785.454339] LNet: 202032:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8786.469538] LNet: Removed LNI 192.168.206.144@tcp [ 8787.235688] Key type .llcrypt unregistered [ 8787.237187] Key type ._llcrypt unregistered [ 8808.685594] Key type ._llcrypt registered [ 8808.686575] Key type .llcrypt registered [ 8808.783612] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8811.164602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8811.280050] alg: No test for adler32 (adler32-zlib) [ 8812.524048] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8812.913634] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8812.916279] LNet: Accept secure, port 988 [ 8814.687420] Key type lgssc registered [ 8815.793765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8821.634381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8821.649395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8823.000292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8825.803756] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8831.772498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8831.837992] Lustre: 203964:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8832.015047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8834.710758] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8839.294886] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8842.960890] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8847.748259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8847.881685] Lustre: 204847:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8848.068537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8851.839176] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8856.032683] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8867.565093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 8877.305295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8877.353182] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8877.394196] Lustre: 205777:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8877.494928] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8881.097413] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8885.278560] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8887.175956] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8898.678281] Lustre: Permanently deactivating lustre-OST0001 [ 8898.688865] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 8905.231564] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 8909.350545] Lustre: Permanently reactivating lustre-OST0001 [ 8909.359308] Lustre: Modifying parameter lustre-OST0001-osc.osc.active in log lustre-client [ 8909.362146] Lustre: Skipped 2 previous similar messages [ 8915.467705] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8915.473061] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 8915.476585] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8915.481509] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.206.144@tcp (at 0@lo) [ 8915.500474] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8920.546602] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8920.547208] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8920.549809] LustreError: Skipped 1 previous similar message [ 8920.554496] Lustre: Skipped 1 previous similar message [ 8920.556165] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8920.560173] Lustre: Skipped 1 previous similar message [ 8925.385129] Lustre: server umount lustre-OST0001 complete [ 8925.670666] LustreError: 204851:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8925.677673] LustreError: 204851:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 8928.224468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8928.227041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.231642] Lustre: Skipped 1 previous similar message [ 8928.234377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8929.248650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8929.253693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8930.788336] LustreError: 204853:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8930.794077] LustreError: 204853:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 8935.904687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8935.906955] Lustre: Skipped 2 previous similar messages [ 8935.909682] LustreError: 205416:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8935.916030] LustreError: 205416:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 8941.029137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8941.030908] LustreError: 204853:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8941.031930] Lustre: Skipped 2 previous similar messages [ 8941.038378] LustreError: 204853:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 8942.047132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8942.147879] Lustre: server umount lustre-OST0000 complete [ 8944.609392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8944.612253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.615885] Lustre: Skipped 1 previous similar message [ 8950.786706] Lustre: server umount lustre-MDT0000 complete [ 8951.265565] LustreError: 203267:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8951.272666] LustreError: 203267:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 8953.185868] LustreError: 203247:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501745 with bad export cookie 11305926902589092495 [ 8953.190329] LustreError: 203247:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8953.190544] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8953.401212] Lustre: server umount lustre-MDT0001 complete [ 8966.906094] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8969.870950] Key type lgssc unregistered [ 8970.216755] LNet: 207865:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8971.237529] LNet: Removed LNI 192.168.206.144@tcp [ 8972.075408] Key type .llcrypt unregistered [ 8972.076924] Key type ._llcrypt unregistered [ 8995.514548] Key type ._llcrypt registered [ 8995.516036] Key type .llcrypt registered [ 8995.614514] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 9011.122247] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 9012.603590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9012.675454] alg: No test for adler32 (adler32-zlib) [ 9013.783734] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 9014.017587] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 9014.019994] LNet: Accept secure, port 988 [ 9015.655242] Key type lgssc registered [ 9016.492062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9023.540171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9028.717138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9033.572099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9038.931721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9043.434697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9043.484531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9043.494049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9044.713719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9044.741838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9044.793857] Lustre: lustre-MDT0000: new disk, initializing [ 9044.851916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9044.864900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9047.477535] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9054.681304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9054.746352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9054.838725] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9054.854138] Lustre: 212557:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9054.885218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9054.888067] Lustre: Skipped 1 previous similar message [ 9054.937598] Lustre: lustre-MDT0001: new disk, initializing [ 9055.015585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9055.044592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9055.048317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9057.525812] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9063.331913] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9066.814887] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9071.446045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9071.510413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9071.688485] Lustre: 213506:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9071.710624] Lustre: lustre-OST0000: new disk, initializing [ 9071.714678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9071.767455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9075.246952] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9077.277181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9077.282251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9077.361784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9081.168931] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9084.306810] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9084.515461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9087.183265] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9087.372210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9092.577423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9092.578706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9092.580753] LustreError: Skipped 1 previous similar message [ 9092.585318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9092.588222] Lustre: Skipped 1 previous similar message [ 9094.802973] Lustre: server umount lustre-OST0000 complete [ 9097.696926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9097.697661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9097.701099] Lustre: Skipped 2 previous similar messages [ 9097.705617] Lustre: Skipped 1 previous similar message [ 9102.817364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9102.819708] Lustre: Skipped 1 previous similar message [ 9103.470951] Lustre: server umount lustre-MDT0000 complete [ 9105.809475] LustreError: 211788:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501897 with bad export cookie 5980549192646520703 [ 9105.813904] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9105.814434] LustreError: 211788:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9106.025783] Lustre: server umount lustre-MDT0001 complete [ 9114.163352] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 06:05:05 (1743501905) [ 9118.654977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9118.960015] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9121.469151] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9124.321095] LustreError: 215798:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9124.327522] LustreError: 215798:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 9127.014556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9127.092075] Lustre: 216478:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9127.248381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9129.669660] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9134.024399] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9137.432548] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9142.135715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9142.249523] Lustre: 217359:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9142.347503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9145.970399] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9150.380364] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9153.361926] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 9157.515668] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 9167.001966] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 9176.531753] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 9197.774983] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 9197.777309] Lustre: Skipped 2 previous similar messages [ 9236.063364] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 9236.066370] Lustre: Skipped 4 previous similar messages [ 9307.947595] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 9307.950114] Lustre: Skipped 8 previous similar messages [ 9322.976814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9322.979781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9322.985440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9327.075151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9327.080348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9327.083281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9328.770688] Lustre: server umount lustre-OST0000 complete [ 9332.195117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9332.195185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9332.200831] Lustre: Skipped 1 previous similar message [ 9337.313099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9337.316020] Lustre: Skipped 4 previous similar messages [ 9337.594940] Lustre: server umount lustre-MDT0000 complete [ 9340.159485] LustreError: 218066:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743502132 with bad export cookie 5980549192646521585 [ 9340.164162] LustreError: 218066:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9340.164346] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9340.410934] Lustre: server umount lustre-MDT0001 complete [ 9355.558642] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 9358.774423] Key type lgssc unregistered [ 9359.118849] LNet: 220182:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9360.165722] LNet: Removed LNI 192.168.206.144@tcp [ 9361.010615] Key type .llcrypt unregistered [ 9361.012247] Key type ._llcrypt unregistered [ 9388.290890] Key type ._llcrypt registered [ 9388.292501] Key type .llcrypt registered [ 9388.382099] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 9390.379068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9390.495647] alg: No test for adler32 (adler32-zlib) [ 9391.771145] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 9392.059045] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 9392.062737] LNet: Accept secure, port 988 [ 9393.839315] Key type lgssc registered [ 9394.810554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9400.346960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9400.360114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9401.678317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9404.208775] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9410.135562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9410.216612] Lustre: 222260:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9410.405848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9413.086125] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9417.874947] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9421.794180] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9426.715753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9426.847851] Lustre: 223142:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.144@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9426.968980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9430.699052] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9435.437072] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9456.613071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9456.613605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9456.619586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9456.623815] Lustre: Skipped 1 previous similar message [ 9461.732831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9461.735546] Lustre: Skipped 1 previous similar message [ 9462.782233] Lustre: server umount lustre-OST0000 complete [ 9466.850711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9466.851176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9466.851995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9466.857576] Lustre: Skipped 2 previous similar messages [ 9471.547762] Lustre: server umount lustre-MDT0000 complete [ 9471.969493] LustreError: 222281:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9471.974896] LustreError: 222281:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 9474.023467] LustreError: 221543:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743502265 with bad export cookie 2708457545685949108 [ 9474.028310] LustreError: 221543:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9474.028698] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9474.228340] Lustre: server umount lustre-MDT0001 complete [ 9489.930808] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 9493.067704] Key type lgssc unregistered [ 9493.434000] LNet: 224918:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9494.502803] LNet: Removed LNI 192.168.206.144@tcp [ 9495.376408] Key type .llcrypt unregistered [ 9495.377703] Key type ._llcrypt unregistered [ 9518.054333] Key type ._llcrypt registered [ 9518.055944] Key type .llcrypt registered [ 9518.168108] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 9518.485020] Key type .llcrypt unregistered [ 9518.486444] Key type ._llcrypt unregistered [ 9539.393122] Key type ._llcrypt registered [ 9539.394378] Key type .llcrypt registered [ 9539.503195] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 9557.270690] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 9559.217939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9559.409978] alg: No test for adler32 (adler32-zlib) [ 9560.731903] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 9561.154367] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 9561.157785] LNet: Accept secure, port 988 [ 9562.927350] Key type lgssc registered [ 9563.923738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9573.137768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9580.542520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9585.997114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9591.632442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9593.420331] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8868 sec ========== 06:13:04 (1743502384) [ 9595.422550] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:13:06 (1743502386) === [ 9599.339574] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:13:10 (1743502390) === [ 9632.673267] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 9635.550570] Key type lgssc unregistered [ 9635.855619] LNet: 231020:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9635.865630] LNet: Removed LNI 192.168.206.144@tcp [ 9636.658497] Key type .llcrypt unregistered [ 9636.660453] Key type ._llcrypt unregistered