[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 484755610 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002253] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007607] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011113] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013031] SELinux: Initializing. [ 0.014061] *** VALIDATE selinux *** [ 0.022826] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027335] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031065] *** VALIDATE tmpfs *** [ 0.032400] *** VALIDATE proc *** [ 0.033235] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.036215] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039035] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043077] debug: unmapping init [mem 0xffffffff9c859000-0xffffffff9c860fff] [ 0.044565] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046361] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054322] rcu: Hierarchical SRCU implementation. [ 0.056430] smp: Bringing up secondary CPUs ... [ 0.057622] x86: Booting SMP configuration: [ 0.058036] .... node #0, CPUs: #1 #2 #3 [ 0.061158] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131758] node 0 deferred pages initialised in 65ms [ 0.135125] devtmpfs: initialized [ 0.136185] x86/mm: Memory block size: 128MB [ 0.138234] gcov: version magic: 0x41383552 [ 0.140391] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142069] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143355] pinctrl core: initialized pinctrl subsystem [ 0.146232] [ 0.146802] ************************************************************* [ 0.149016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151012] ** ** [ 0.153013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155013] ** ** [ 0.157013] ** This means that this kernel is built to expose internal ** [ 0.160015] ** IOMMU data structures, which may compromise security on ** [ 0.162013] ** your system. ** [ 0.163012] ** ** [ 0.165014] ** If you see this message and you are not debugging the ** [ 0.167017] ** kernel, report this immediately to your vendor! ** [ 0.169015] ** ** [ 0.171012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173012] ************************************************************* [ 0.175708] NET: Registered protocol family 16 [ 0.178332] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.181039] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.183549] cpuidle: using governor menu [ 0.184713] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186449] PCI: Using configuration type 1 for base access [ 0.188109] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194093] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197075] cryptd: max_cpu_qlen set to 1000 [ 0.199155] ACPI: Added _OSI(Module Device) [ 0.199986] ACPI: Added _OSI(Processor Device) [ 0.201008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.201897] ACPI: Added _OSI(Processor Aggregator Device) [ 0.203946] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.208364] ACPI: Interpreter enabled [ 0.209046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210020] ACPI: Using IOAPIC for interrupt routing [ 0.212120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216445] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226385] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.228051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.230031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.238436] acpiphp: Slot [2] registered [ 0.240132] acpiphp: Slot [5] registered [ 0.242142] acpiphp: Slot [6] registered [ 0.243177] acpiphp: Slot [7] registered [ 0.245163] acpiphp: Slot [8] registered [ 0.246174] acpiphp: Slot [9] registered [ 0.248143] acpiphp: Slot [10] registered [ 0.250134] acpiphp: Slot [3] registered [ 0.251104] acpiphp: Slot [4] registered [ 0.252114] acpiphp: Slot [11] registered [ 0.254142] acpiphp: Slot [12] registered [ 0.255104] acpiphp: Slot [13] registered [ 0.256145] acpiphp: Slot [14] registered [ 0.258086] acpiphp: Slot [15] registered [ 0.259091] acpiphp: Slot [16] registered [ 0.260108] acpiphp: Slot [17] registered [ 0.262128] acpiphp: Slot [18] registered [ 0.263071] acpiphp: Slot [19] registered [ 0.264063] acpiphp: Slot [20] registered [ 0.265097] acpiphp: Slot [21] registered [ 0.266101] acpiphp: Slot [22] registered [ 0.268114] acpiphp: Slot [23] registered [ 0.270122] acpiphp: Slot [24] registered [ 0.271143] acpiphp: Slot [25] registered [ 0.272119] acpiphp: Slot [26] registered [ 0.274139] acpiphp: Slot [27] registered [ 0.275104] acpiphp: Slot [28] registered [ 0.277108] acpiphp: Slot [29] registered [ 0.278097] acpiphp: Slot [30] registered [ 0.279065] acpiphp: Slot [31] registered [ 0.281067] PCI host bridge to bus 0000:00 [ 0.281960] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.283014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.285018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.286018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.288015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.289015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.290157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.292941] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.294514] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.304017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.308471] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.311020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.313020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.316021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.317959] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.320997] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.323045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.325695] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.330018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.342015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.348021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.354016] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.360020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.366017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.387020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.403769] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.416017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.423028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.445022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.454865] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.464018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.473021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.492018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.502802] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.513017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.526016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.540019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.549710] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.557017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.563021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.576016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.584084] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.590020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.595018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.611017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.620870] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.623201] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.625386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.627300] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.628134] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.631107] iommu: Default domain type: Passthrough [ 0.632295] SCSI subsystem initialized [ 0.634133] ACPI: bus type USB registered [ 0.635074] usbcore: registered new interface driver usbfs [ 0.636048] usbcore: registered new interface driver hub [ 0.637054] usbcore: registered new device driver usb [ 0.638125] pps_core: LinuxPPS API ver. 1 registered [ 0.639007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.641051] PTP clock support registered [ 0.643039] EDAC MC: Ver: 3.0.0 [ 0.644093] PCI: Using ACPI for IRQ routing [ 0.645432] NetLabel: Initializing [ 0.646005] NetLabel: domain hash size = 128 [ 0.646862] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.647139] NetLabel: unlabeled traffic allowed by default [ 0.648080] vgaarb: loaded [ 0.649207] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.650009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.657175] clocksource: Switched to clocksource kvm-clock [ 0.754441] VFS: Disk quotas dquot_6.6.0 [ 0.755854] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.758950] *** VALIDATE ramfs *** [ 0.760264] *** VALIDATE hugetlbfs *** [ 0.762400] pnp: PnP ACPI init [ 0.764853] pnp: PnP ACPI: found 6 devices [ 0.781320] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.784658] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.786628] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.788818] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.791233] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.793702] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.796553] NET: Registered protocol family 2 [ 0.799047] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.803850] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.807365] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.812406] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.814869] TCP: Hash tables configured (established 65536 bind 65536) [ 0.817328] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.820122] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.822724] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.825513] NET: Registered protocol family 1 [ 0.827930] RPC: Registered named UNIX socket transport module. [ 0.829924] RPC: Registered udp transport module. [ 0.831703] RPC: Registered tcp transport module. [ 0.833306] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.835477] NET: Registered protocol family 44 [ 0.837104] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.839149] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.841114] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.843350] PCI: CLS 0 bytes, default 64 [ 0.845681] Unpacking initramfs... [ 2.234445] debug: unmapping init [mem 0xffff9acabcc54000-0xffff9acabffbffff] [ 2.238442] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.239935] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.242637] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.746075] Initialise system trusted keyrings [ 2.747494] Key type blacklist registered [ 2.748967] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.757341] zbud: loaded [ 2.760292] *** VALIDATE nfs *** [ 2.761577] *** VALIDATE nfs4 *** [ 2.763038] pstore: using deflate compression [ 2.765949] Platform Keyring initialized [ 2.860964] NET: Registered protocol family 38 [ 2.862519] Key type asymmetric registered [ 2.864248] Asymmetric key parser 'x509' registered [ 2.865925] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.869123] io scheduler mq-deadline registered [ 2.870635] io scheduler kyber registered [ 2.872310] io scheduler bfq registered [ 2.873951] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.876832] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.879165] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.881509] ACPI: Power Button [PWRF] [ 2.886552] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.892783] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.903314] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.909664] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.921630] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.951380] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.979799] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.984426] Non-volatile memory driver v1.3 [ 2.985715] Linux agpgart interface v0.103 [ 3.012737] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.015573] vda: detected capacity change from 0 to 76677120 [ 3.030308] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.033155] vdb: detected capacity change from 0 to 1073741824 [ 3.046720] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.048844] vdc: detected capacity change from 0 to 2621440000 [ 3.063683] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.065503] vdd: detected capacity change from 0 to 2621440000 [ 3.078167] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.080388] vde: detected capacity change from 0 to 4294967296 [ 3.102145] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.105355] vdf: detected capacity change from 0 to 4294967296 [ 3.112621] libphy: Fixed MDIO Bus: probed [ 3.117313] usbcore: registered new interface driver usbserial_generic [ 3.118764] usbserial: USB Serial support registered for generic [ 3.120453] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.123250] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.124390] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.125812] mousedev: PS/2 mouse device common for all mice [ 3.130531] rtc_cmos 00:05: RTC can wake from S4 [ 3.132174] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.135163] rtc_cmos 00:05: registered as rtc0 [ 3.137901] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.139487] intel_pstate: CPU model not supported [ 3.140903] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.144431] hid: raw HID events driver (C) Jiri Kosina [ 3.144587] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.145898] usbcore: registered new interface driver usbhid [ 3.149271] usbhid: USB HID core driver [ 3.150266] drop_monitor: Initializing network drop monitor service [ 3.152154] Initializing XFRM netlink socket [ 3.153606] NET: Registered protocol family 10 [ 3.155357] Segment Routing with IPv6 [ 3.156344] NET: Registered protocol family 17 [ 3.157866] mpls_gso: MPLS GSO support [ 3.161659] RAS: Correctable Errors collector initialized. [ 3.162958] AVX version of gcm_enc/dec engaged. [ 3.163834] AES CTR mode by8 optimization enabled [ 3.212474] sched_clock: Marking stable (3212456510, 0)->(4125363004, -912906494) [ 3.214544] registered taskstats version 1 [ 3.216053] Loading compiled-in X.509 certificates [ 3.217096] zswap: loaded using pool lzo/zbud [ 3.235076] Key type big_key registered [ 3.243395] Key type encrypted registered [ 3.244544] ima: No TPM chip found, activating TPM-bypass! [ 3.245747] ima: Allocated hash algorithm: sha1 [ 3.246817] ima: No architecture policies found [ 3.247745] evm: Initialising EVM extended attributes: [ 3.248704] evm: security.selinux [ 3.249364] evm: security.ima [ 3.249892] evm: security.capability [ 3.250567] evm: HMAC attrs: 0x1 [ 3.251958] rtc_cmos 00:05: setting system clock to 2026-09-05 23:32:09 UTC (1788651129) [ 3.255878] debug: unmapping init [mem 0xffffffff9d803000-0xffffffff9d9fffff] [ 3.257994] debug: unmapping init [mem 0xffffffff9c582000-0xffffffff9c858fff] [ 3.265049] Write protecting the kernel read-only data: 28672k [ 3.267778] debug: unmapping init [mem 0xffffffff9ac03000-0xffffffff9adfffff] [ 3.269479] debug: unmapping init [mem 0xffffffff9b514000-0xffffffff9b5fffff] [ 3.293548] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.299123] systemd[1]: Detected virtualization kvm. [ 3.300287] systemd[1]: Detected architecture x86-64. [ 3.301318] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.324976] systemd[1]: No hostname configured. [ 3.326162] systemd[1]: Set hostname to . [ 3.327538] random: systemd: uninitialized urandom read (16 bytes read) [ 3.329552] systemd[1]: Initializing machine ID from random generator. [ 3.446959] random: systemd: uninitialized urandom read (16 bytes read) [ 3.449644] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.454074] random: systemd: uninitialized urandom read (16 bytes read) [ 3.456137] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.460994] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.009197] device-mapper: uevent: version 1.0.3 [ 4.011319] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 4.519266] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.667619] virtio_net virtio0 ens2: renamed from eth0 [ 4.719589] scsi host0: ata_piix [ 4.749587] scsi host1: ata_piix [ 4.752861] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.755304] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.681886] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.767388] random: crng init done [ 9.769258] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.060766] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.506718] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.241796] SELinux: Disabled at runtime. [ 14.321230] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.343105] systemd[1]: Detected virtualization kvm. [ 14.349267] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.765755] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.771725] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.785401] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.793279] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.798410] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.811857] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.825391] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped targ[ 16.310145] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 17.163568] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.635817] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.737994] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.099772] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.255346] EDAC sbridge: Ver: 1.1.2 [ 22.953733] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 23.630282] NFS: Registering the id_resolver key type [ 23.636851] Key type id_resolver registered [ 23.641402] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg137-server login: [ 87.063266] hrtimer: interrupt took 4498892 ns [ 90.784582] libcfs: loading out-of-tree module taints kernel. [ 90.873292] Key type ._llcrypt registered [ 90.876640] Key type .llcrypt registered [ 91.054389] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 110.988439] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 112.904283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 112.921582] alg: No test for adler32 (adler32-zlib) [ 114.491317] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 115.733143] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 117.559250] Key type lgssc registered [ 119.690257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 138.811519] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 184.728560] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 198.853608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.899408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.185489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 200.216199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 200.301631] Lustre: lustre-MDT0000: new disk, initializing [ 200.366881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.380274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.008690] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.340969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.469534] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.507918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.516504] Lustre: Skipped 1 previous similar message [ 220.596171] Lustre: lustre-MDT0001: new disk, initializing [ 220.654457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.673930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.688385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.710482] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.493493] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 241.690947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.983384] Lustre: lustre-OST0000: new disk, initializing [ 241.993051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 242.012104] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.087517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.874446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.888815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 243.000649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.996740] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.838133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 263.988640] Lustre: lustre-OST0001: new disk, initializing [ 263.994139] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 264.000560] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.090142] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.919420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 269.936383] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 269.980674] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 271.159577] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 283.935610] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 290.133699] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.569985] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing check_logdir /tmp/testlogs/ [ 302.190987] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing yml_node [ 307.149215] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 310.039784] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 312.885869] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 314.675215] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 19:37:20 EDT 2026 [ 332.665653] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 334.169102] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 352.224815] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.227531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.249832] Lustre: Skipped 3 previous similar messages [ 352.270929] Lustre: Skipped 3 previous similar messages [ 356.314785] Lustre: server umount lustre-MDT0000 complete [ 360.097695] LustreError: 10339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651486 with bad export cookie 11880106023613240739 [ 360.099556] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.104378] LustreError: 10339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 362.465472] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.467504] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.473267] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 362.478954] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 362.519940] Lustre: Skipped 1 previous similar message [ 365.471632] LustreError: 9532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.471949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.485967] Lustre: Skipped 2 previous similar messages [ 366.650607] Lustre: server umount lustre-MDT0001 complete [ 370.699478] Lustre: server umount lustre-OST0000 complete [ 374.425762] Lustre: server umount lustre-OST0001 complete [ 381.587266] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 392.491563] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 433.449863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.804259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 433.862914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 433.984316] Lustre: lustre-MDT0000: new disk, initializing [ 434.113710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.137620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 439.133276] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 451.393777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.573720] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 451.651879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 451.655559] Lustre: Skipped 1 previous similar message [ 451.749450] Lustre: lustre-MDT0001: new disk, initializing [ 451.890526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 451.946489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 451.961201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 456.779378] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 466.218971] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 472.098930] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 478.839879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 479.314326] Lustre: lustre-OST0000: new disk, initializing [ 479.324972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 479.345302] Lustre: 17048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 479.491771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.118319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 481.131569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 481.339883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 487.274304] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 498.434438] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 503.980853] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 504.204590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 509.173405] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 509.385283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 511.968634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 511.977517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.992883] LustreError: Skipped 1 previous similar message [ 511.995317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.023685] Lustre: Skipped 1 previous similar message [ 517.105566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 517.116658] Lustre: Skipped 2 previous similar messages [ 517.653764] Lustre: server umount lustre-OST0000 complete [ 522.209944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.232866] Lustre: Skipped 1 previous similar message [ 527.329994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 527.343120] Lustre: Skipped 3 previous similar messages [ 528.678308] Lustre: server umount lustre-MDT0000 complete [ 532.449139] LustreError: 15945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.463740] LustreError: 15945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 532.751598] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651658 with bad export cookie 11880106023613242265 [ 532.756934] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 532.764135] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 533.220987] Lustre: server umount lustre-MDT0001 complete [ 537.186847] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 19:41:02 (1788651662) [ 545.143084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.768779] LustreError: 19315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.871617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 550.886049] LustreError: 19316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.061318] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 560.864985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.120806] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 566.573418] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 575.344755] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 581.757555] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 590.991839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 602.501223] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 612.150493] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 621.536869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 621.548334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.556959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 621.565126] Lustre: Skipped 1 previous similar message [ 621.580634] Lustre: Skipped 1 previous similar message [ 626.659300] LustreError: 21135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.674400] LustreError: 21135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 626.775558] Lustre: server umount lustre-OST0000 complete [ 631.776924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.279677] Lustre: server umount lustre-MDT0000 complete [ 640.970857] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651767 with bad export cookie 11880106023613243154 [ 640.983397] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.988035] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 641.286354] Lustre: server umount lustre-MDT0001 complete [ 651.925456] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 654.978499] Key type lgssc unregistered [ 655.296209] LNet: 22848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.318863] LNetError: 22848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.359416] LNet: Removed LNI 192.168.201.137@tcp [ 656.366319] Key type .llcrypt unregistered [ 656.369535] Key type ._llcrypt unregistered [ 681.316548] Key type ._llcrypt registered [ 681.318373] Key type .llcrypt registered [ 681.477857] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 682.607495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 682.723342] alg: No test for adler32 (adler32-zlib) [ 683.851342] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 684.132145] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 685.896425] Key type lgssc registered [ 686.967956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 694.556300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 694.587116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.139603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 700.434160] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 706.530749] LustreError: 24260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.554955] LustreError: 24260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 710.594812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.115853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 715.764914] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 724.292357] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 731.285346] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 739.416687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 739.944821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 740.983186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 746.986278] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 754.394667] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 772.078506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 772.079809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 772.093771] LustreError: Skipped 1 previous similar message [ 772.105852] Lustre: Skipped 1 previous similar message [ 772.112337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 772.121983] Lustre: Skipped 1 previous similar message [ 775.780180] Lustre: server umount lustre-OST0000 complete [ 782.313291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 782.324529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 782.326109] Lustre: Skipped 1 previous similar message [ 782.343330] Lustre: Skipped 1 previous similar message [ 785.763465] Lustre: server umount lustre-MDT0000 complete [ 787.426185] LustreError: 24256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 787.461587] LustreError: 24256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 790.510529] LustreError: 24242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651916 with bad export cookie 11583952424152954750 [ 790.521360] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 790.527368] LustreError: 24242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 790.923459] Lustre: server umount lustre-MDT0001 complete [ 802.830883] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 806.178811] Key type lgssc unregistered [ 806.570648] LNet: 28236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.580824] LNetError: 28236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.595666] LNet: Removed LNI 192.168.201.137@tcp [ 807.572168] Key type .llcrypt unregistered [ 807.574088] Key type ._llcrypt unregistered [ 833.537520] Key type ._llcrypt registered [ 833.539486] Key type .llcrypt registered [ 833.701163] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 835.672305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 835.712273] alg: No test for adler32 (adler32-zlib) [ 836.908031] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 837.482602] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 839.295273] Key type lgssc registered [ 840.848390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 851.386276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 851.409351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.781941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.476099] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 862.187943] LustreError: 29649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 862.230829] LustreError: 29649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 867.300442] LustreError: 29650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 867.829245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.209308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 873.995307] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 883.050981] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 889.315489] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 909.618970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 910.013496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.896483] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 922.354640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 926.269406] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 932.841848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 932.845732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.846883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 932.869765] Lustre: Skipped 1 previous similar message [ 937.363094] Lustre: server umount lustre-OST0000 complete [ 943.080562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.094917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.101384] Lustre: Skipped 1 previous similar message [ 943.121417] Lustre: Skipped 2 previous similar messages [ 947.572545] Lustre: server umount lustre-MDT0000 complete [ 948.198902] LustreError: 32441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.251273] LustreError: 32441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 952.061614] LustreError: 29630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652078 with bad export cookie 18263599196650859120 [ 952.073372] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 952.079628] LustreError: 29630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 952.513225] Lustre: server umount lustre-MDT0001 complete [ 965.949965] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 968.466198] Key type lgssc unregistered [ 968.684864] LNet: 33530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.690854] LNetError: 33530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.704555] LNet: Removed LNI 192.168.201.137@tcp [ 969.461601] Key type .llcrypt unregistered [ 969.463686] Key type ._llcrypt unregistered [ 993.417516] Key type ._llcrypt registered [ 993.419913] Key type .llcrypt registered [ 993.577666] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 995.114539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 995.141112] alg: No test for adler32 (adler32-zlib) [ 996.313511] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 996.669428] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 998.367172] Key type lgssc registered [ 999.929775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1008.655494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1008.689190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1010.149513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.362899] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1020.387148] LustreError: 34938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.405338] LustreError: 34938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1023.441308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.738896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1028.727916] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1036.435502] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1042.750640] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1049.362094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1049.711954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1054.832829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1056.665860] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1066.088664] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1075.680693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1075.700585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.738623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1078.768526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1078.773223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1078.779207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.731142] Lustre: server umount lustre-OST0000 complete [ 1088.995432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.015475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1089.018081] Lustre: Skipped 1 previous similar message [ 1092.055740] Lustre: server umount lustre-MDT0000 complete [ 1094.114088] LustreError: 34935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.145659] LustreError: 34935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1096.455182] LustreError: 35673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652222 with bad export cookie 9443412879761184263 [ 1096.466326] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1096.468259] LustreError: 35673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1096.782932] Lustre: server umount lustre-MDT0001 complete [ 1108.286745] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1110.730152] Key type lgssc unregistered [ 1110.966651] LNet: 38472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.976982] LNetError: 38472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.009128] LNet: Removed LNI 192.168.201.137@tcp [ 1111.919436] Key type .llcrypt unregistered [ 1111.921492] Key type ._llcrypt unregistered [ 1141.109895] Key type ._llcrypt registered [ 1141.112642] Key type .llcrypt registered [ 1141.230179] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1142.035052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1142.186163] alg: No test for adler32 (adler32-zlib) [ 1143.230352] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 1143.439573] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1145.080102] Key type lgssc registered [ 1146.785938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1155.500107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1155.515113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1157.090484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.169609] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1167.331695] LustreError: 39884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.352261] LustreError: 39884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1171.637103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1172.024866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1177.163468] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1185.462444] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1191.524546] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1198.779365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1199.160735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.253629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1207.660819] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1216.448130] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1222.639421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1222.641540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1222.651683] LustreError: Skipped 1 previous similar message [ 1222.654272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1222.673203] Lustre: Skipped 1 previous similar message [ 1223.164179] Lustre: lustre-OST0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 1223.170217] Lustre: Skipped 1 previous similar message [ 1227.752419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1227.765554] Lustre: Skipped 1 previous similar message [ 1228.558326] Lustre: server umount lustre-OST0000 complete [ 1248.741473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1248.761037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.783387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1248.788203] Lustre: Skipped 1 previous similar message [ 1252.898991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.900542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.812747] Lustre: server umount lustre-MDT0000 complete [ 1257.953184] LustreError: 41724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.992311] LustreError: 41724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1258.486839] LustreError: 39865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652384 with bad export cookie 9130890451943967478 [ 1258.489137] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1258.499802] LustreError: 39865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1265.291699] Lustre: server umount lustre-MDT0001 complete [ 1277.456146] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1280.304329] Key type lgssc unregistered [ 1280.548334] LNet: 43592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.567380] LNetError: 43592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1281.651506] LNet: Removed LNI 192.168.201.137@tcp [ 1282.537961] Key type .llcrypt unregistered [ 1282.540072] Key type ._llcrypt unregistered [ 1305.473217] Key type ._llcrypt registered [ 1305.475129] Key type .llcrypt registered [ 1305.557156] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1306.844209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1306.865555] alg: No test for adler32 (adler32-zlib) [ 1307.915730] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 1308.211518] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1309.976706] Key type lgssc registered [ 1311.422975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1320.313507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1320.332979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1321.699470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.255205] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1331.170131] LustreError: 45002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1331.191928] LustreError: 45002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1336.290109] LustreError: 45001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.492154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.125980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1342.816798] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1351.351413] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1357.395503] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1364.218260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1364.508664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.603240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1370.216326] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1379.099428] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1389.538822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1389.553956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.571102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1391.105920] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 1393.127836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.130781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1394.384639] Lustre: server umount lustre-MDT0000 complete [ 1396.184291] LustreError: 44996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1397.714544] LustreError: 44981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652523 with bad export cookie 17045148361100350541 [ 1397.724299] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1397.728913] LustreError: 44981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1398.241029] LustreError: 44997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.241192] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.292808] Lustre: Skipped 1 previous similar message [ 1398.300995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1398.303808] Lustre: Skipped 2 previous similar messages [ 1402.272708] LustreError: 44996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.289026] LustreError: 44996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1406.416179] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 1406.427732] Lustre: Skipped 2 previous similar messages [ 1411.533809] LustreError: 47783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1411.541102] LustreError: 47783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1412.063512] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1412.517796] Lustre: server umount lustre-MDT0001 complete [ 1428.180415] Lustre: server umount lustre-OST0000 complete [ 1447.530189] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1449.944972] Key type lgssc unregistered [ 1450.306541] LNet: 48860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.325647] LNetError: 48860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.344788] LNet: Removed LNI 192.168.201.137@tcp [ 1451.464406] Key type .llcrypt unregistered [ 1451.467363] Key type ._llcrypt unregistered [ 1479.585963] Key type ._llcrypt registered [ 1479.594886] Key type .llcrypt registered [ 1479.723232] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1481.591508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1481.645117] alg: No test for adler32 (adler32-zlib) [ 1482.793091] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 1483.095068] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1484.808050] Key type lgssc registered [ 1485.768905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1492.991312] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1493.008847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1508.511986] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1513.631619] LustreError: 50246:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 1540.383428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.260502] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1555.352847] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1720.981856] Lustre: server umount lustre-OST0000 complete [ 1737.856694] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1740.109407] Key type lgssc unregistered [ 1740.341239] LNet: 51933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.345282] LNetError: 51933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.361227] LNet: Removed LNI 192.168.201.137@tcp [ 1741.232941] Key type .llcrypt unregistered [ 1741.239552] Key type ._llcrypt unregistered [ 1764.644342] Key type ._llcrypt registered [ 1764.646557] Key type .llcrypt registered [ 1764.762708] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1765.676517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1765.716082] alg: No test for adler32 (adler32-zlib) [ 1767.070584] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 1767.391406] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1769.127466] Key type lgssc registered [ 1770.533516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1778.777517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1778.795703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.216379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.018662] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1790.433339] LustreError: 53342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.448310] LustreError: 53342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1793.695545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.063699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1798.804209] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1807.593441] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1813.760609] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1821.074424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1821.472406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.595941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1827.460298] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1834.792915] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1838.951350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.956225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1838.981979] Lustre: Skipped 1 previous similar message [ 1844.206132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.318956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.321981] Lustre: Skipped 1 previous similar message [ 1851.360323] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1851.633792] Lustre: server umount lustre-OST0000 complete [ 1855.458083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1855.465344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.480351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1859.553362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.556325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1859.574283] Lustre: Skipped 1 previous similar message [ 1863.830860] Lustre: server umount lustre-MDT0000 complete [ 1864.675056] LustreError: 54077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1864.695779] LustreError: 54077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1867.628662] LustreError: 53322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652993 with bad export cookie 18425246363293925927 [ 1867.634555] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1867.637395] LustreError: 53322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1868.030369] Lustre: server umount lustre-MDT0001 complete [ 1879.814124] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1883.268851] Key type lgssc unregistered [ 1883.644704] LNet: 56877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.657748] LNetError: 56877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.673484] LNet: Removed LNI 192.168.201.137@tcp [ 1884.564995] Key type .llcrypt unregistered [ 1884.568620] Key type ._llcrypt unregistered [ 1906.303696] Key type ._llcrypt registered [ 1906.306088] Key type .llcrypt registered [ 1906.496373] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1907.860734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1908.135170] alg: No test for adler32 (adler32-zlib) [ 1909.219635] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 1909.631526] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1911.367385] Key type lgssc registered [ 1912.561705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1919.781890] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1919.796784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1935.327552] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1940.447698] LustreError: 58241:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 1967.135213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.993693] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1980.432331] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1987.118402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.472605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.528422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1991.277237] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1992.673628] LustreError: 58266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.689043] LustreError: 58266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1997.793498] LustreError: 58267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.283294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.963291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2005.715688] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2014.725618] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2021.779514] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2026.468304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2026.490461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.510397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2028.002095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2028.002929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.129228] Lustre: server umount lustre-OST0000 complete [ 2057.186262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2057.198780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.211690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.214419] Lustre: Skipped 1 previous similar message [ 2062.096964] Lustre: server umount lustre-MDT0000 complete [ 2063.856921] LustreError: 59275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.875703] LustreError: 59275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2066.378830] LustreError: 58248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653192 with bad export cookie 8460679654048524037 [ 2066.412810] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2072.867739] Lustre: server umount lustre-MDT0001 complete [ 2084.063715] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2087.054697] Key type lgssc unregistered [ 2087.409581] LNet: 61981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.416428] LNetError: 61981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.486327] LNet: Removed LNI 192.168.201.137@tcp [ 2089.307234] Key type .llcrypt unregistered [ 2089.312635] Key type ._llcrypt unregistered [ 2113.779137] Key type ._llcrypt registered [ 2113.780774] Key type .llcrypt registered [ 2113.943178] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2115.927688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2116.031940] alg: No test for adler32 (adler32-zlib) [ 2117.279249] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 2117.480755] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2119.127178] Key type lgssc registered [ 2120.438835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2129.952221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2129.993898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.572107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.633554] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2141.168990] LustreError: 63395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.193542] LustreError: 63395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2145.971699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.296197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2151.085876] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2160.532587] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2167.942706] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2175.658217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2175.976253] LustreError: 65219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2176.231376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2181.622960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2183.392267] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2192.995730] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2211.305694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.334963] Lustre: Skipped 1 previous similar message [ 2211.336442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2216.431072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2216.446184] Lustre: Skipped 2 previous similar messages [ 2216.810384] Lustre: server umount lustre-OST0000 complete [ 2221.537493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.553219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2221.557039] Lustre: Skipped 1 previous similar message [ 2226.659179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2226.670033] Lustre: Skipped 1 previous similar message [ 2227.616433] Lustre: server umount lustre-MDT0000 complete [ 2231.197068] LustreError: 63377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653357 with bad export cookie 1778106442415149583 [ 2231.204260] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2231.212293] LustreError: 63377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2231.638794] Lustre: server umount lustre-MDT0001 complete [ 2245.928131] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2250.506974] Key type lgssc unregistered [ 2250.873044] LNet: 66950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.877280] LNetError: 66950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2250.902082] LNet: Removed LNI 192.168.201.137@tcp [ 2252.288893] Key type .llcrypt unregistered [ 2252.290586] Key type ._llcrypt unregistered [ 2282.000657] Key type ._llcrypt registered [ 2282.003671] Key type .llcrypt registered [ 2282.119931] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2283.213261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2283.270892] alg: No test for adler32 (adler32-zlib) [ 2284.798557] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 2285.004259] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2286.703176] Key type lgssc registered [ 2288.139253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2299.121251] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 20:10:24 (1788653424) [ 2306.900944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2306.940814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.461106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2313.890053] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2317.798181] LustreError: 68652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.835894] LustreError: 68652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2322.916101] LustreError: 68651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.780515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.207923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2328.466550] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2337.528869] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2344.059888] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2351.521178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2351.993197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2353.066675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2359.959628] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2369.488630] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2376.732436] Lustre: Failing over lustre-MDT0000 [ 2377.152271] Lustre: server umount lustre-MDT0000 complete [ 2377.726657] LustreError: 68646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.744903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.750911] LustreError: 68648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.759576] Lustre: Skipped 2 previous similar messages [ 2379.782753] LustreError: 68648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2384.865986] LustreError: 68652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.892717] LustreError: 68652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2387.238241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.407723] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2387.533630] Lustre: *** cfs_fail_loc=135, val=0*** [ 2387.643749] LustreError: 71739:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2387.651691] LustreError: 71739:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.137@tcp: cfg command failed: rc = -2 [ 2387.659697] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2387.659697] [ 2387.670137] LustreError: MGC192.168.201.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2387.698516] LustreError: 71727:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2387.710167] LustreError: 71727:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2387.720301] LustreError: 71727:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2387.732704] LustreError: 71727:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2387.768144] Lustre: server umount lustre-MDT0000 complete [ 2387.773688] LustreError: 71727:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2393.055514] LustreError: 68646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2393.081926] LustreError: 68646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2395.271524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.495612] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2395.507205] LustreError: 68643:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2395.842764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.910881] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2398.159185] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2401.264046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2401.331966] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2401.403623] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2401.408822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2410.477674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2410.492537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.506560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2411.493851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2411.513169] Lustre: Skipped 1 previous similar message [ 2416.044378] Lustre: server umount lustre-OST0000 complete [ 2421.734848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.737038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2421.746320] Lustre: Skipped 2 previous similar messages [ 2421.748385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2426.521994] Lustre: server umount lustre-MDT0000 complete [ 2426.854739] LustreError: 68651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.871260] LustreError: 68651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2430.613946] LustreError: 70495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653556 with bad export cookie 2157160167777548143 [ 2430.637782] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.645128] LustreError: 70495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2431.180651] Lustre: server umount lustre-MDT0001 complete [ 2445.610308] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2449.265711] Key type lgssc unregistered [ 2449.592514] LNet: 73520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2449.598677] LNetError: 73520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2449.631123] LNet: Removed LNI 192.168.201.137@tcp [ 2450.696761] Key type .llcrypt unregistered [ 2450.698525] Key type ._llcrypt unregistered [ 2477.991179] Key type ._llcrypt registered [ 2477.996380] Key type .llcrypt registered [ 2478.169706] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2479.849986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2479.944095] alg: No test for adler32 (adler32-zlib) [ 2481.352961] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 2481.631499] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2483.351365] Key type lgssc registered [ 2484.600063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2493.842942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2493.860599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.588786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2500.606945] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2505.698593] LustreError: 74932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2505.736525] LustreError: 74932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2510.130931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.501858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2515.712807] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2524.496411] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2530.103669] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2537.205798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2537.660903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2540.756858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2544.506748] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2553.168651] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2562.370913] Lustre: Failing over lustre-MDT0000 [ 2562.798257] Lustre: server umount lustre-MDT0000 complete [ 2565.097670] LustreError: 74928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.604062] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.630164] Lustre: Skipped 1 previous similar message [ 2570.217790] LustreError: 74926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.251382] LustreError: 74926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2574.399044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.638992] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2574.906252] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2575.039339] LustreError: 78018:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2575.051783] LustreError: 78018:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.137@tcp: cfg command failed: rc = -9 [ 2575.068757] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2575.068757] [ 2575.078391] LustreError: MGC192.168.201.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2575.105569] LustreError: 78007:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2575.128075] LustreError: 78007:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2575.144157] LustreError: 78007:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2575.154666] LustreError: 78007:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2575.188657] Lustre: server umount lustre-MDT0000 complete [ 2575.197714] LustreError: 78007:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2575.332932] LustreError: 74926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.380552] LustreError: 74926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2580.440651] LustreError: 76223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.465244] LustreError: 76223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2584.257729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.503404] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2584.511992] LustreError: 74924:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2584.890527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.952605] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2585.364713] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2590.186293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2590.211512] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2590.249990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2590.647199] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2600.928116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2600.942213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.957369] Lustre: Skipped 1 previous similar message [ 2600.960600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2601.439947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2605.546699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.555925] Lustre: Skipped 2 previous similar messages [ 2607.179690] Lustre: server umount lustre-OST0000 complete [ 2615.777263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.790431] Lustre: Skipped 1 previous similar message [ 2615.797324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2615.804742] Lustre: Skipped 1 previous similar message [ 2618.408282] Lustre: server umount lustre-MDT0000 complete [ 2620.903809] LustreError: 74932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.935681] LustreError: 74932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2622.389698] LustreError: 74913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653748 with bad export cookie 16077704798642167828 [ 2622.405665] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2622.807092] Lustre: server umount lustre-MDT0001 complete [ 2635.334357] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2638.122344] Key type lgssc unregistered [ 2638.418682] LNet: 79798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2638.424938] LNetError: 79798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2638.438466] LNet: Removed LNI 192.168.201.137@tcp [ 2639.192124] Key type .llcrypt unregistered [ 2639.194957] Key type ._llcrypt unregistered [ 2654.166241] Key type ._llcrypt registered [ 2654.170639] Key type .llcrypt registered [ 2654.297238] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2654.697760] Key type .llcrypt unregistered [ 2654.704680] Key type ._llcrypt unregistered [ 2679.931410] Key type ._llcrypt registered [ 2679.933977] Key type .llcrypt registered [ 2680.041766] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2681.200730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2681.279217] alg: No test for adler32 (adler32-zlib) [ 2682.347170] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 2682.545061] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2684.171200] Key type lgssc registered [ 2685.371888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2693.560992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2693.581783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.224121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2700.222063] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2705.377418] LustreError: 81697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.394942] LustreError: 81697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2710.498463] LustreError: 81698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.592505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2711.047935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2716.535653] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2723.467435] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2729.551917] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2737.619934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2737.968739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2740.012436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2744.599912] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2753.272897] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2770.912837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2770.929927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.969638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.941856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.950924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2771.951609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2774.327347] Lustre: server umount lustre-OST0000 complete [ 2782.181832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.188932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.198388] Lustre: Skipped 1 previous similar message [ 2782.222986] Lustre: Skipped 2 previous similar messages [ 2784.384086] Lustre: server umount lustre-MDT0000 complete [ 2787.298175] LustreError: 81694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.330644] LustreError: 81694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2788.195365] LustreError: 82431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653914 with bad export cookie 17408265392385205287 [ 2788.197880] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2788.206743] LustreError: 82431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2788.546259] Lustre: server umount lustre-MDT0001 complete [ 2800.236431] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2803.522323] Key type lgssc unregistered [ 2803.912066] LNet: 85266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2803.920423] LNetError: 85266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2803.942636] LNet: Removed LNI 192.168.201.137@tcp [ 2804.903349] Key type .llcrypt unregistered [ 2804.904630] Key type ._llcrypt unregistered [ 2831.027658] Key type ._llcrypt registered [ 2831.029892] Key type .llcrypt registered [ 2831.194442] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2832.358983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2832.506531] alg: No test for adler32 (adler32-zlib) [ 2833.691475] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 2834.013942] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2835.695362] Key type lgssc registered [ 2837.006291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2845.182517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2845.207919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.786707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.273187] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2855.909954] LustreError: 86673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.946680] LustreError: 86673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2861.029822] LustreError: 86674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2861.351289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.613961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2867.032201] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2875.515208] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2881.208943] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2888.173873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2888.449785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.524254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2894.766975] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2903.195501] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2916.322069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2916.329067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.343064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2916.354465] Lustre: Skipped 1 previous similar message [ 2916.378719] Lustre: Skipped 1 previous similar message [ 2919.393493] Lustre: server umount lustre-OST0000 complete [ 2926.564231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.573126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2931.691253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2931.706154] Lustre: Skipped 2 previous similar messages [ 2935.777819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2935.792741] Lustre: Skipped 1 previous similar message [ 2938.336059] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2938.598552] Lustre: server umount lustre-MDT0000 complete [ 2940.897564] LustreError: 88131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.910440] LustreError: 88131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2942.539969] LustreError: 86654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654068 with bad export cookie 7938931620338084544 [ 2942.559438] LustreError: 86654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2942.561044] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2943.087995] Lustre: server umount lustre-MDT0001 complete [ 2955.812669] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2959.257334] Key type lgssc unregistered [ 2959.566437] LNet: 90207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2959.576911] LNetError: 90207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2959.591229] LNet: Removed LNI 192.168.201.137@tcp [ 2960.966924] Key type .llcrypt unregistered [ 2960.969964] Key type ._llcrypt unregistered [ 2991.431875] Key type ._llcrypt registered [ 2991.433735] Key type .llcrypt registered [ 2991.551807] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2993.200774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2993.215567] alg: No test for adler32 (adler32-zlib) [ 2994.250575] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 2994.589923] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2996.351248] Key type lgssc registered [ 2997.832716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3006.125038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3006.158215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3007.678379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3012.496843] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3017.189356] LustreError: 91617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.212241] LustreError: 91617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3019.866149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.237884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3023.845664] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3032.181458] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3037.808558] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3044.115120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3044.547609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3051.421193] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3052.750254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3060.398600] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3078.650830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.651757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3078.659546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3078.673292] Lustre: Skipped 1 previous similar message [ 3083.047624] Lustre: server umount lustre-OST0000 complete [ 3088.873252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.880103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3088.891609] Lustre: Skipped 1 previous similar message [ 3088.903541] Lustre: Skipped 1 previous similar message [ 3093.985945] LustreError: 91614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.007362] LustreError: 91614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3095.419329] Lustre: server umount lustre-MDT0000 complete [ 3099.108807] LustreError: 91617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.120482] LustreError: 91617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3099.438828] LustreError: 91598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654225 with bad export cookie 1397810686967451335 [ 3099.445917] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3099.452308] LustreError: 91598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3099.858510] Lustre: server umount lustre-MDT0001 complete [ 3111.923968] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3114.664274] Key type lgssc unregistered [ 3114.967197] LNet: 95171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3114.974465] LNetError: 95171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3114.994273] LNet: Removed LNI 192.168.201.137@tcp [ 3115.711800] Key type .llcrypt unregistered [ 3115.717410] Key type ._llcrypt unregistered [ 3135.981982] Key type ._llcrypt registered [ 3135.984353] Key type .llcrypt registered [ 3136.087425] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3137.013376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3137.162181] alg: No test for adler32 (adler32-zlib) [ 3138.172888] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 3138.385600] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3140.079188] Key type lgssc registered [ 3141.325142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3148.418218] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3148.443956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3164.000668] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3169.119339] LustreError: 96531:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 3195.807257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3201.995344] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3209.431844] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3214.664110] Lustre: server umount lustre-OST0000 complete [ 3223.963258] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 20:25:49 (1788654349) [ 3232.449399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.940735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3238.370763] LustreError: 98172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.393112] LustreError: 98172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3239.533635] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3243.490537] LustreError: 98171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3248.609209] LustreError: 98172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3249.653904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.108177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3254.212297] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3262.364804] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3269.362832] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3276.792318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3277.087357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.154789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3282.911632] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3291.516368] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3309.029711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3309.053316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.069334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3311.076783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3311.080562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.532829] Lustre: server umount lustre-OST0000 complete [ 3316.706314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3316.717235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.731215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.738297] Lustre: Skipped 1 previous similar message [ 3321.317567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3321.325468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3322.630693] Lustre: server umount lustre-MDT0000 complete [ 3326.438478] LustreError: 98171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3326.466142] LustreError: 98171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3327.146800] LustreError: 98151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654453 with bad export cookie 9402943874152144961 [ 3327.150584] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3327.160576] LustreError: 98151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3327.896622] Lustre: server umount lustre-MDT0001 complete [ 3339.869182] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3343.338246] Key type lgssc unregistered [ 3343.554833] LNet: 101862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3343.563263] LNetError: 101862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3343.573974] LNet: Removed LNI 192.168.201.137@tcp [ 3344.234542] Key type .llcrypt unregistered [ 3344.236515] Key type ._llcrypt unregistered [ 3389.693505] Key type ._llcrypt registered [ 3389.696593] Key type .llcrypt registered [ 3389.832274] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3390.909759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3391.065952] alg: No test for adler32 (adler32-zlib) [ 3392.450556] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 3392.830361] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3394.583175] Key type lgssc registered [ 3395.794272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3404.263505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3404.316788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3405.817085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3410.438557] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3416.034093] LustreError: 104046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3416.048323] LustreError: 104046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3419.912402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3420.479516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3424.801726] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3434.705234] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3442.869208] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3451.365139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3451.648723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3457.883723] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3466.868681] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3476.217305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3486.689026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3486.695063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.703653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3486.714160] Lustre: Skipped 1 previous similar message [ 3491.816475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3491.834477] Lustre: Skipped 1 previous similar message [ 3492.461966] Lustre: server umount lustre-OST0000 complete [ 3502.049038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3502.070606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3502.082561] Lustre: Skipped 2 previous similar messages [ 3504.470965] Lustre: server umount lustre-MDT0000 complete [ 3507.175671] LustreError: 104045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3507.238974] LustreError: 104045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3508.589475] LustreError: 104027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654634 with bad export cookie 8944193338631567545 [ 3508.594798] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3508.599768] LustreError: 104027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3508.841957] Lustre: server umount lustre-MDT0001 complete [ 3519.765136] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3523.222654] Key type lgssc unregistered [ 3523.591944] LNet: 107600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3523.601151] LNetError: 107600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3523.621724] LNet: Removed LNI 192.168.201.137@tcp [ 3524.788955] Key type .llcrypt unregistered [ 3524.790725] Key type ._llcrypt unregistered [ 3545.111155] Key type ._llcrypt registered [ 3545.112965] Key type .llcrypt registered [ 3545.250267] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3546.749069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3546.820047] alg: No test for adler32 (adler32-zlib) [ 3548.003777] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 3548.432336] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3550.220297] Key type lgssc registered [ 3551.631470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3561.235954] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3561.255592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3576.802995] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3581.919958] LustreError: 108840:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 3608.607205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3616.432400] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3624.700839] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3632.131648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.366983] LustreError: MGC192.168.201.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3632.376810] LustreError: 109863:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3632.382546] LustreError: 109863:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3632.387623] LustreError: 109863:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3632.439314] Lustre: server umount lustre-MDT0000 complete [ 3632.441370] LustreError: 109863:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3652.578108] Lustre: 108176:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788654762/real 1788654762] req@00000000e11185dc x1875540363447680/t0(0) o400->MGC192.168.201.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788654778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3652.614977] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3652.626807] LustreError: 108852:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3673.238102] Lustre: server umount lustre-OST0000 complete [ 3684.296413] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 3696.886867] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3732.588937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.848479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3732.877910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3732.984357] Lustre: lustre-MDT0000: new disk, initializing [ 3733.040730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.063631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3738.317567] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3750.602053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3750.816561] Lustre: 112661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3750.893663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3750.907676] Lustre: Skipped 1 previous similar message [ 3751.088680] Lustre: lustre-MDT0001: new disk, initializing [ 3751.216548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3751.254939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3751.263214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3756.133208] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3766.224389] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3772.510738] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3779.698878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3780.221139] Lustre: lustre-OST0000: new disk, initializing [ 3780.230196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3780.236680] Lustre: 114630:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3780.376908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3786.384323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3786.412885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3786.648400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3788.637323] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3801.537939] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3807.925824] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3808.176302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3814.595060] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3814.790183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3817.442701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3817.445624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3817.455166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3817.485273] Lustre: Skipped 1 previous similar message [ 3822.562920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3822.571585] Lustre: Skipped 2 previous similar messages [ 3827.684652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.704531] Lustre: Skipped 1 previous similar message [ 3831.263495] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3831.458461] Lustre: server umount lustre-OST0000 complete [ 3837.922351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.932275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3837.939117] Lustre: Skipped 1 previous similar message [ 3837.957484] Lustre: Skipped 1 previous similar message [ 3843.044913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.059068] Lustre: Skipped 1 previous similar message [ 3850.207172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3850.417597] Lustre: server umount lustre-MDT0000 complete [ 3854.830695] LustreError: 112655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654981 with bad export cookie 8672311008647727361 [ 3854.838836] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3854.856076] LustreError: 112655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3855.343683] Lustre: server umount lustre-MDT0001 complete [ 3865.769380] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 20:36:30 (1788654990) [ 3870.196690] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3888.679766] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 3899.343112] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3941.471547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3941.708383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3941.755821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3941.942501] Lustre: lustre-MDT0000: new disk, initializing [ 3942.067745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3942.097837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3948.049811] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3959.687720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3959.851412] Lustre: 119535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3959.911843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3959.916177] Lustre: Skipped 1 previous similar message [ 3960.046340] Lustre: lustre-MDT0001: new disk, initializing [ 3960.195221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3960.243192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3960.255934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3965.520217] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3975.590740] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3982.343947] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3989.846585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3990.195250] Lustre: lustre-OST0000: new disk, initializing [ 3990.200222] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3990.206127] Lustre: 121502:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3990.344993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3997.133725] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3998.283790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3998.286884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3998.386748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4007.259845] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4013.196688] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4013.409493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4018.193024] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4018.451467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4024.306931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4024.315336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.319340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4024.319351] Lustre: Skipped 2 previous similar messages [ 4024.321336] LustreError: Skipped 1 previous similar message [ 4024.363671] Lustre: Skipped 1 previous similar message [ 4035.039151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4035.307279] Lustre: server umount lustre-OST0000 complete [ 4039.649355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4044.769326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4044.776799] Lustre: Skipped 8 previous similar messages [ 4045.885717] Lustre: server umount lustre-MDT0000 complete [ 4049.909684] LustreError: 122497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4049.947951] LustreError: 122497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4050.169455] LustreError: 120328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655176 with bad export cookie 8672311008647728243 [ 4050.177423] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4050.190062] LustreError: 120328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4050.657888] Lustre: server umount lustre-MDT0001 complete [ 4061.031945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4061.559419] LustreError: 123724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4061.622397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4066.671557] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4066.786669] LustreError: 123725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.884428] LustreError: 123724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4076.011757] LustreError: 123725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4076.811918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4082.728466] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4091.766310] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4097.666578] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4105.500486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4113.948734] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4123.415441] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4130.600898] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4137.379748] Lustre: server umount lustre-OST0000 complete [ 4143.079325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.090325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4143.098460] Lustre: Skipped 2 previous similar messages [ 4143.121851] Lustre: Skipped 1 previous similar message [ 4154.015343] Lustre: 108176:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788655264/real 1788655264] req@000000007b748702 x1875540363568512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788655280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4154.056931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4155.103164] Lustre: 108178:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788655264/real 1788655264] req@00000000670dcea5 x1875540363568896/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788655280 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4156.383131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4156.550613] Lustre: server umount lustre-MDT0000 complete [ 4159.463139] Lustre: 108175:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788655269/real 1788655269] req@000000007e2d9e20 x1875540363570560/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788655285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4159.470784] LustreError: 123719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.537406] LustreError: 123719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4160.402699] LustreError: 125570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655286 with bad export cookie 8672311008647729132 [ 4160.412646] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4160.417970] LustreError: 125570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4160.661443] Lustre: server umount lustre-MDT0001 complete [ 4173.526251] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4177.141665] Key type lgssc unregistered [ 4177.527920] LNet: 127354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4177.538731] LNetError: 127354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4177.557330] LNet: Removed LNI 192.168.201.137@tcp [ 4178.637444] Key type .llcrypt unregistered [ 4178.639945] Key type ._llcrypt unregistered [ 4198.292456] Key type ._llcrypt registered [ 4198.293669] Key type .llcrypt registered [ 4198.358476] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4217.691789] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4220.199690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4220.336665] alg: No test for adler32 (adler32-zlib) [ 4221.636504] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 4222.076504] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4223.967171] Key type lgssc registered [ 4225.675967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4267.404866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4267.443367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4268.737273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4268.776207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4268.841543] Lustre: lustre-MDT0000: new disk, initializing [ 4268.913563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.948444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4274.598694] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4285.502237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.602728] Lustre: 130501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4285.656986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4285.660762] Lustre: Skipped 1 previous similar message [ 4285.722187] Lustre: lustre-MDT0001: new disk, initializing [ 4285.850354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4285.914349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4285.921556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4290.365144] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4300.650771] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4307.113294] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4314.676944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4315.020303] Lustre: lustre-OST0000: new disk, initializing [ 4315.023512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4315.027857] Lustre: 132473:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4315.121471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4321.889957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4321.903539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4321.998247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4322.136838] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4331.337427] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4336.403170] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4336.642574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4341.784975] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4342.245064] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4347.872698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4347.874307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.881985] LustreError: Skipped 1 previous similar message [ 4347.882564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4347.899817] Lustre: Skipped 1 previous similar message [ 4352.992749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4353.005200] Lustre: Skipped 2 previous similar messages [ 4358.114435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4358.120335] Lustre: Skipped 1 previous similar message [ 4359.135548] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4359.466948] Lustre: server umount lustre-OST0000 complete [ 4367.845724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4367.856823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.880050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4370.452457] Lustre: server umount lustre-MDT0000 complete [ 4374.499612] LustreError: 130496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655500 with bad export cookie 16886095249316479756 [ 4374.502798] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4374.522742] LustreError: 130496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4375.115550] Lustre: server umount lustre-MDT0001 complete [ 4384.136424] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 20:45:09 (1788655509) [ 4391.503363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4392.011366] LustreError: 134888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.023716] LustreError: 134888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4392.071712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.545466] LustreError: 134889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4398.147913] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4402.666299] LustreError: 134888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.780989] LustreError: 134889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.359076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.749717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4414.393580] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4422.700933] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4428.440425] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4433.378411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.380611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4433.398665] Lustre: Skipped 2 previous similar messages [ 4433.412627] Lustre: Skipped 3 previous similar messages [ 4437.232605] Lustre: server umount lustre-MDT0000 complete [ 4439.521066] LustreError: 134888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4439.536210] LustreError: 134888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4441.150903] LustreError: 134868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655567 with bad export cookie 16886095249316480638 [ 4441.161994] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4441.873880] Lustre: server umount lustre-MDT0001 complete [ 4451.244634] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 20:46:16 (1788655576) [ 4458.680023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4474.335811] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4479.458439] LustreError: 137304:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 4505.184107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4513.630941] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4526.295171] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4529.025371] Lustre: server umount lustre-OST0000 complete [ 4540.307915] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 20:47:45 (1788655665) [ 4547.614364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4548.136849] LustreError: 138758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4548.230803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4554.114982] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4565.290831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.774467] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4580.609483] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4587.667694] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4596.039759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4596.341740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4596.351300] Lustre: Skipped 1 previous similar message [ 4604.397088] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4613.121789] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4627.427111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4627.438055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.468601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4627.481895] Lustre: Skipped 2 previous similar messages [ 4631.182445] Lustre: server umount lustre-MDT0000 complete [ 4633.568500] LustreError: 138758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.584472] LustreError: 138758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4635.072309] LustreError: 138738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655761 with bad export cookie 16886095249316481303 [ 4635.085850] LustreError: 138738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4635.087817] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4635.452592] Lustre: server umount lustre-MDT0001 complete [ 4649.120395] Lustre: server umount lustre-OST0000 complete [ 4659.015254] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 20:49:44 (1788655784) [ 4666.140920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4666.620540] LustreError: 142415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4666.686996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.137946] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4680.948695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.925924] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4695.655474] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4702.787651] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4711.969406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4718.668665] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4727.036205] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4732.566114] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4735.871864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4736.122704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4740.930315] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4741.172196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4746.209047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4746.252125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4746.285482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4746.317779] Lustre: Skipped 3 previous similar messages [ 4746.364399] Lustre: Skipped 4 previous similar messages [ 4750.265259] Lustre: server umount lustre-OST0000 complete [ 4756.414660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.422254] Lustre: Skipped 1 previous similar message [ 4762.806588] Lustre: server umount lustre-MDT0000 complete [ 4766.691272] LustreError: 142410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4766.716572] LustreError: 142410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4768.609825] LustreError: 142395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655894 with bad export cookie 16886095249316482374 [ 4768.612134] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4768.619639] LustreError: 142395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4768.875328] Lustre: server umount lustre-MDT0001 complete [ 4781.895564] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 20:51:47 (1788655907) [ 4792.405995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4808.292209] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4813.407695] LustreError: 146599:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 4839.135426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.143548] Lustre: Skipped 2 previous similar messages [ 4845.785439] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4853.446776] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4859.576831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.986941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4863.638034] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4871.996860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4877.323392] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4885.143450] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4890.982755] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4897.207118] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4897.585339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4903.798573] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4904.113594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4908.009254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4908.016621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4908.030449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4908.034882] Lustre: Skipped 4 previous similar messages [ 4920.287306] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4920.520618] Lustre: server umount lustre-MDT0000 complete [ 4924.457860] LustreError: 146605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656050 with bad export cookie 16886095249316483396 [ 4924.466702] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4924.471401] LustreError: 146605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4924.906931] Lustre: server umount lustre-MDT0001 complete [ 4937.958795] Lustre: server umount lustre-OST0000 complete [ 4948.395610] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 20:54:33 (1788656073) [ 4956.337922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4971.999875] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4977.119497] LustreError: 150794:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 5009.872566] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5017.213428] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5025.446345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.084624] LustreError: 150819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.094609] LustreError: 150819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5026.263608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5031.533275] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5039.977284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5045.105684] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5054.518747] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5062.886674] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5072.070632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5072.270662] Lustre: lustre-OST0001: new disk, initializing [ 5072.279608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5072.290713] Lustre: 153673:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5080.189390] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5080.632639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5080.642064] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5080.730985] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5090.980353] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5096.276685] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5096.516327] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5101.505913] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5101.820306] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5106.148400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5106.155912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5106.159847] Lustre: Skipped 3 previous similar messages [ 5106.177329] Lustre: Skipped 10 previous similar messages [ 5110.050749] Lustre: server umount lustre-OST0000 complete [ 5119.513946] Lustre: server umount lustre-OST0001 complete [ 5138.399254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5138.643840] Lustre: server umount lustre-MDT0000 complete [ 5143.633166] LustreError: 152576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656269 with bad export cookie 16886095249316484348 [ 5143.653676] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5143.657487] LustreError: 152576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5144.282743] Lustre: server umount lustre-MDT0001 complete [ 5171.154278] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 20:58:16 (1788656296) [ 5172.937118] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5174.873636] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5177.153656] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 20:58:22 (1788656302) [ 5184.445738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5184.701460] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5184.732146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5184.960982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.978522] Lustre: Skipped 6 previous similar messages [ 5191.104268] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5200.850685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5201.011539] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5201.104248] Lustre: 157233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5206.999845] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5215.882512] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5222.535755] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5231.203177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5231.449217] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5238.897821] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5241.466559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5247.903303] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5253.432825] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5253.628995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5258.831492] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5259.119828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5261.792891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5261.800891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5261.813873] Lustre: Skipped 4 previous similar messages [ 5261.821490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5261.828158] Lustre: Skipped 8 previous similar messages [ 5275.103151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5275.588346] Lustre: server umount lustre-OST0000 complete [ 5300.657700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5302.134481] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.163532] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5302.164212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.799369] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5316.006908] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5323.143564] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5323.506429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5329.676131] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5329.964631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5338.753253] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cd8c5523000.ost_server_uuid 50 [ 5340.894020] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9cd8c5523000.ost_server_uuid in IDLE state after 0 sec [ 5341.083511] LustreError: 161445:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.37@tcp) returned error from blocking AST (req@0000000077e021a9 x1875541070468352 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000008f482459/0xea577629b2eb9e56 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.37@tcp remote: 0x46193096abf70f08 expref: 5 pid: 157240 timeout: 5441 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5341.145675] LustreError: lustre-MDT0000: A client on nid 192.168.201.37@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5341.164317] LustreError: 157230:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.37@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000008f482459/0xea577629b2eb9e56 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.37@tcp remote: 0x46193096abf70f08 expref: 6 pid: 157240 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5348.334199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5348.346591] LustreError: Skipped 1 previous similar message [ 5354.158171] Lustre: server umount lustre-OST0000 complete [ 5372.384187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5372.609848] Lustre: server umount lustre-MDT0000 complete [ 5373.921317] LustreError: 161445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.933887] LustreError: 161445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5376.546789] LustreError: 157227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656502 with bad export cookie 16886095249316485531 [ 5376.548374] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5376.564377] LustreError: 157227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5383.065640] Lustre: server umount lustre-MDT0001 complete [ 5395.573217] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5398.714868] Key type lgssc unregistered [ 5399.047844] LNet: 163363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5399.058870] LNetError: 163363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5399.072916] LNet: Removed LNI 192.168.201.137@tcp [ 5400.228919] Key type .llcrypt unregistered [ 5400.230528] Key type ._llcrypt unregistered [ 5424.638720] Key type ._llcrypt registered [ 5424.640660] Key type .llcrypt registered [ 5424.726880] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5425.818437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5425.956906] alg: No test for adler32 (adler32-zlib) [ 5426.955642] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 5427.226804] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5428.895285] Key type lgssc registered [ 5430.489643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5438.757215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5438.776404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5440.318709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5445.399657] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5450.730635] LustreError: 164776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5450.759981] LustreError: 164776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5455.847866] LustreError: 164777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5455.936146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5456.358579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5462.213654] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5470.731908] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5477.468550] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5485.040549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5485.471144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5492.894658] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5501.498611] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5510.077930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5518.859333] Lustre: Failing over lustre-MDT0000 [ 5519.087898] Lustre: server umount lustre-MDT0000 complete [ 5520.354981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5520.360767] LustreError: 167280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5520.363628] Lustre: Skipped 2 previous similar messages [ 5520.378858] LustreError: 167280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5525.476821] LustreError: 164776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.504788] LustreError: 164776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5530.594643] LustreError: 164777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5530.612181] LustreError: 164777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5531.696732] LustreError: 164758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.37@tcp arrived at 1788656657 with bad export cookie 13457685720489575908 [ 5531.719031] LustreError: 164758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5536.736484] Lustre: 164086:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788656646/real 1788656646] req@00000000308c5ebf x1875542333748480/t0(0) o400->MGC192.168.201.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788656662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5536.773606] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5539.555759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.860048] LustreError: 164771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5541.881593] LustreError: 164771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5546.975975] LustreError: 164084:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000044dc4893 x1875542333753856/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5547.352502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5547.395927] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5552.499811] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5552.613696] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5552.634552] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5554.666785] Lustre: lustre-MDT0000: Denying connection for new client 0a4c3b9c-dbb9-4eb5-a8f4-79cc95a64352 (at 192.168.201.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5559.766757] Lustre: lustre-MDT0000: Denying connection for new client 0a4c3b9c-dbb9-4eb5-a8f4-79cc95a64352 (at 192.168.201.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 5567.460368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5567.469858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5567.484690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5573.096606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5573.112171] Lustre: Skipped 2 previous similar messages [ 5577.183188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5577.430878] Lustre: server umount lustre-OST0000 complete [ 5581.553712] LustreError: 168535:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5581.560356] Lustre: 167883:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5581.567720] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5581.581161] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5581.592355] LustreError: 167883:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000a07b71d x1875542333770240/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5581.604187] LustreError: 167883:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5581.611216] LustreError: 167883:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5581.620226] Lustre: lustre-MDT0000: Recovery over after 0:29, of 2 clients 0 recovered and 2 were evicted. [ 5581.915269] Lustre: server umount lustre-MDT0000 complete [ 5583.330296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5583.344093] Lustre: Skipped 1 previous similar message [ 5583.350621] LustreError: 166066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.367368] LustreError: 166066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5585.465041] LustreError: 164756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656711 with bad export cookie 13457685720489576230 [ 5585.481173] LustreError: 164756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5585.489892] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5585.491849] LustreError: 164084:0:(import.c:751:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5587.958988] Lustre: server umount lustre-MDT0001 complete [ 5600.554397] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5603.595318] Key type lgssc unregistered [ 5603.931780] LNet: 169286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5603.949521] LNetError: 169286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5603.970253] LNet: Removed LNI 192.168.201.137@tcp [ 5604.868826] Key type .llcrypt unregistered [ 5604.878341] Key type ._llcrypt unregistered [ 5631.923961] Key type ._llcrypt registered [ 5631.924995] Key type .llcrypt registered [ 5632.053743] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5633.054413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5633.100239] alg: No test for adler32 (adler32-zlib) [ 5634.206682] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 5634.460918] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5636.151166] Key type lgssc registered [ 5637.440750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5645.292398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5645.312326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5646.670075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5651.118340] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5656.050368] LustreError: 170694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5656.085401] LustreError: 170694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5660.372425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.787369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5666.075595] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5673.489247] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5679.941433] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5686.785555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5687.174971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5691.251670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5693.753600] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5702.093164] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5710.829664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5710.831676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5710.838679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5710.844072] Lustre: Skipped 1 previous similar message [ 5710.853615] Lustre: Skipped 1 previous similar message [ 5711.237979] Lustre: server umount lustre-OST0000 complete [ 5715.940435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5715.955361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5715.967876] Lustre: Skipped 1 previous similar message [ 5715.979520] Lustre: Skipped 1 previous similar message [ 5721.057205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5721.067241] Lustre: Skipped 1 previous similar message [ 5726.176352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5726.179530] Lustre: Skipped 1 previous similar message [ 5729.248034] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5729.433109] Lustre: server umount lustre-MDT0000 complete [ 5731.300350] LustreError: 170694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5731.320190] LustreError: 170694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5732.941435] LustreError: 170673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656859 with bad export cookie 6465826669425279337 [ 5732.961624] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5733.330594] Lustre: server umount lustre-MDT0001 complete [ 5746.622963] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5749.508530] Key type lgssc unregistered [ 5749.764721] LNet: 174226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5749.775955] LNetError: 174226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5749.791158] LNet: Removed LNI 192.168.201.137@tcp [ 5750.720354] Key type .llcrypt unregistered [ 5750.723053] Key type ._llcrypt unregistered [ 5784.600529] Key type ._llcrypt registered [ 5784.603251] Key type .llcrypt registered [ 5784.728746] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5786.513839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5786.569136] alg: No test for adler32 (adler32-zlib) [ 5787.643490] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 5787.996692] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5789.735289] Key type lgssc registered [ 5791.073990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5798.560320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5798.572054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.966067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5804.360381] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5810.149017] LustreError: 175926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5810.176543] LustreError: 175926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5813.033349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.576735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5818.803504] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5827.997631] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5833.844560] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5841.267902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5841.762164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5844.853197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5849.429525] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5858.076199] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5865.440440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5865.448322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5865.477821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5868.518996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5868.523981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5868.538048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5870.508954] Lustre: server umount lustre-OST0000 complete [ 5875.172760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5875.189981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5875.210658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5875.213377] Lustre: Skipped 1 previous similar message [ 5878.756552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5878.757386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5880.820504] Lustre: server umount lustre-MDT0000 complete [ 5883.880672] LustreError: 175921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.904545] LustreError: 175921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5884.603569] LustreError: 175908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657010 with bad export cookie 17558843191432067000 [ 5884.618684] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5884.931158] Lustre: server umount lustre-MDT0001 complete [ 5896.937133] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5899.997304] Key type lgssc unregistered [ 5900.300852] LNet: 179459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5900.310788] LNetError: 179459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5900.334074] LNet: Removed LNI 192.168.201.137@tcp [ 5901.325151] Key type .llcrypt unregistered [ 5901.327469] Key type ._llcrypt unregistered [ 5923.231744] Key type ._llcrypt registered [ 5923.232807] Key type .llcrypt registered [ 5923.296452] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5924.270515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5924.302733] alg: No test for adler32 (adler32-zlib) [ 5925.287784] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 5925.539973] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5927.303189] Key type lgssc registered [ 5928.032990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5937.817724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5937.831830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5939.254115] Lustre: *** cfs_fail_loc=135, val=0*** [ 5939.355212] LustreError: 180980:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5939.365797] LustreError: 180980:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.137@tcp: cfg command failed: rc = -2 [ 5939.370715] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5939.370715] [ 5939.380702] LustreError: MGC192.168.201.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5939.402315] LustreError: 180941:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5939.414876] LustreError: 180941:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5939.418925] LustreError: 180941:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5939.424266] LustreError: 180941:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5939.489269] Lustre: server umount lustre-MDT0000 complete [ 5939.490906] LustreError: 180941:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5946.105227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.725706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5951.038765] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5956.074916] LustreError: 181352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.088858] LustreError: 181352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5959.650730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.935462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5964.137885] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5972.151931] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5977.722671] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5980.644119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5980.651066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5980.672056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5981.664313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5981.664961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.762631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5985.765659] Lustre: Skipped 2 previous similar messages [ 5986.130607] Lustre: server umount lustre-MDT0000 complete [ 5989.715661] LustreError: 181332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657115 with bad export cookie 1444461029538442850 [ 5989.720216] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5989.730362] LustreError: 181332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5990.172189] Lustre: server umount lustre-MDT0001 complete [ 6001.943938] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6004.368941] Key type lgssc unregistered [ 6004.716070] LNet: 183671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6004.725549] LNetError: 183671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6005.812249] LNet: Removed LNI 192.168.201.137@tcp [ 6006.705257] Key type .llcrypt unregistered [ 6006.707194] Key type ._llcrypt unregistered [ 6030.661438] Key type ._llcrypt registered [ 6030.665412] Key type .llcrypt registered [ 6030.776681] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6031.005287] Key type .llcrypt unregistered [ 6031.007421] Key type ._llcrypt unregistered [ 6046.712861] Key type ._llcrypt registered [ 6046.719278] Key type .llcrypt registered [ 6046.863868] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6048.125562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6048.197850] alg: No test for adler32 (adler32-zlib) [ 6049.369831] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 6049.661693] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6051.375295] Key type lgssc registered [ 6052.225548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6060.371353] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6060.392095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6075.999603] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6081.119395] LustreError: 185771:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 6107.807532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6113.911572] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6121.156940] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6127.843410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6127.976527] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6127.983518] Lustre: MGS: Client dfc7bbdf-4514-45c0-a326-8c091202ca32 (at 0@lo) reconnecting [ 6127.989969] Lustre: MGC192.168.201.137@tcp: Connection restored to 0@lo (at 0@lo) [ 6128.284491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6128.357049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6132.526567] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6133.728217] LustreError: 185797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6133.742898] LustreError: 185797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6138.849090] LustreError: 185796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6141.700601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.000527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6146.712656] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6153.951147] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6159.647726] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6162.641729] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6177.772116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6177.776610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6177.780583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6177.796592] Lustre: Skipped 1 previous similar message [ 6177.804561] Lustre: Skipped 1 previous similar message [ 6181.777661] Lustre: server umount lustre-OST0000 complete [ 6188.000250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6188.007710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6188.012341] Lustre: Skipped 1 previous similar message [ 6191.632735] Lustre: server umount lustre-MDT0000 complete [ 6193.123252] LustreError: 186810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6193.145382] LustreError: 186810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6195.630623] LustreError: 185777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657321 with bad export cookie 15699593113338102645 [ 6195.635795] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6195.641714] LustreError: 185777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6196.144404] Lustre: server umount lustre-MDT0001 complete [ 6209.771858] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6213.033720] Key type lgssc unregistered [ 6213.334716] LNet: 189788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6213.344840] LNetError: 189788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6213.370802] LNet: Removed LNI 192.168.201.137@tcp [ 6214.531881] Key type .llcrypt unregistered [ 6214.533974] Key type ._llcrypt unregistered [ 6240.496198] Key type ._llcrypt registered [ 6240.498296] Key type .llcrypt registered [ 6240.591527] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6241.485141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6241.649981] alg: No test for adler32 (adler32-zlib) [ 6242.711316] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 6242.922124] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6244.575182] Key type lgssc registered [ 6245.669525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6252.140697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6252.152848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.596801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6258.021572] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6263.779505] LustreError: 191199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6263.804887] LustreError: 191199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6265.400938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.696368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6269.521324] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6275.632167] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6281.161899] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6287.755563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6288.188774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6294.972938] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6299.474935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6303.021631] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6308.708258] Lustre: Failing over lustre-MDT0000 [ 6309.159979] Lustre: server umount lustre-MDT0000 complete [ 6309.865396] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.869854] LustreError: 191195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6309.884773] Lustre: Skipped 2 previous similar messages [ 6309.911045] LustreError: 191195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6310.901305] LustreError: 191195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6314.978605] LustreError: 192287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6315.000243] LustreError: 192287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6320.097931] LustreError: 191199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6320.129073] LustreError: 191199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6326.177093] Lustre: 190511:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788657436/real 1788657436] req@0000000042485ed0 x1875543189381632/t0(0) o400->MGC192.168.201.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788657452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6326.201802] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6328.493245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.366354] LustreError: 192287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6331.377138] LustreError: 192287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6337.865482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6337.932964] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6341.583185] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6342.447252] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6343.161624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6343.175696] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6343.229478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6346.805323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6357.841725] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6371.818104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6371.830923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.847047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6372.321915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6372.741556] Lustre: server umount lustre-OST0000 complete [ 6378.987539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6378.989380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6378.993105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6379.005615] Lustre: Skipped 2 previous similar messages [ 6379.060620] Lustre: Skipped 2 previous similar messages [ 6384.097826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6389.220340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6389.225532] Lustre: Skipped 2 previous similar messages [ 6391.263198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6391.446774] Lustre: server umount lustre-MDT0000 complete [ 6394.336962] LustreError: 191198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6394.348427] LustreError: 191198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6394.537776] LustreError: 191179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657520 with bad export cookie 12884694723952393663 [ 6394.546364] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6395.126265] Lustre: server umount lustre-MDT0001 complete [ 6406.509826] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6409.760631] Key type lgssc unregistered [ 6410.040856] LNet: 196200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6410.049519] LNetError: 196200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6410.069532] LNet: Removed LNI 192.168.201.137@tcp [ 6411.186393] Key type .llcrypt unregistered [ 6411.187858] Key type ._llcrypt unregistered [ 6438.129232] Key type ._llcrypt registered [ 6438.131643] Key type .llcrypt registered [ 6438.226950] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6439.136146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6439.143994] alg: No test for adler32 (adler32-zlib) [ 6440.160499] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 6440.404458] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6442.113661] Key type lgssc registered [ 6442.837361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6450.625924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6450.645122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6452.233494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6457.333723] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6462.434544] LustreError: 197610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6462.485539] LustreError: 197610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6467.561626] LustreError: 197611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6467.698210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6468.128886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6473.162163] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6480.177586] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6485.593859] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6492.755068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6493.121770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6497.308704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6500.372145] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6507.915599] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6511.796920] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6519.366728] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6535.567429] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6543.328217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6543.334637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.341956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6543.360348] Lustre: Skipped 1 previous similar message [ 6543.378647] Lustre: Skipped 1 previous similar message [ 6548.458701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6548.464077] Lustre: Skipped 1 previous similar message [ 6549.024111] Lustre: server umount lustre-OST0000 complete [ 6553.570550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.591680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6553.599952] Lustre: Skipped 1 previous similar message [ 6558.689250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6558.692992] Lustre: Skipped 1 previous similar message [ 6563.809299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6563.812031] Lustre: Skipped 1 previous similar message [ 6567.391545] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6567.637458] Lustre: server umount lustre-MDT0000 complete [ 6568.936994] LustreError: 198338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6568.964303] LustreError: 198338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6571.591849] LustreError: 197591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657697 with bad export cookie 6189946802185307881 [ 6571.596913] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6571.604065] LustreError: 197591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6572.117853] Lustre: server umount lustre-MDT0001 complete [ 6584.391713] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6587.713889] Key type lgssc unregistered [ 6588.103676] LNet: 201486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6588.120562] LNetError: 201486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6588.149854] LNet: Removed LNI 192.168.201.137@tcp [ 6589.442560] Key type .llcrypt unregistered [ 6589.446563] Key type ._llcrypt unregistered [ 6613.265812] Key type ._llcrypt registered [ 6613.273045] Key type .llcrypt registered [ 6613.435795] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6614.763866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6615.116119] alg: No test for adler32 (adler32-zlib) [ 6616.329343] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 6616.596918] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6618.295182] Key type lgssc registered [ 6619.565806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6627.583114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6627.603701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6629.223238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6633.929315] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6639.590849] LustreError: 202752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6639.640557] LustreError: 202752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6644.706895] LustreError: 202751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6645.102239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6645.760237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6652.663536] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6660.154722] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6666.149559] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6673.861330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6674.435033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6675.522428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6682.899391] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6690.989632] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6697.676814] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6709.088542] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6720.081526] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6725.769459] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6740.967061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6740.979780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6740.984179] Lustre: Skipped 1 previous similar message [ 6746.084245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6746.093491] Lustre: Skipped 1 previous similar message [ 6751.211028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6751.221927] Lustre: Skipped 1 previous similar message [ 6752.223156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6752.386029] Lustre: server umount lustre-OST0000 complete [ 6758.369665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6758.379873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6758.392437] Lustre: Skipped 1 previous similar message [ 6758.400708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6761.441524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6763.019266] Lustre: server umount lustre-MDT0000 complete [ 6766.569416] LustreError: 202752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6766.587680] LustreError: 202752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6767.016967] LustreError: 202732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657893 with bad export cookie 1301921238613349508 [ 6767.022284] LustreError: 202732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6767.029303] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6767.435461] Lustre: server umount lustre-MDT0001 complete [ 6782.530741] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6785.394950] Key type lgssc unregistered [ 6785.812701] LNet: 207724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6785.816663] LNetError: 207724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6785.827805] LNet: Removed LNI 192.168.201.137@tcp [ 6786.555596] Key type .llcrypt unregistered [ 6786.560729] Key type ._llcrypt unregistered [ 6810.762542] Key type ._llcrypt registered [ 6810.766160] Key type .llcrypt registered [ 6810.873548] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6811.879132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6811.896689] alg: No test for adler32 (adler32-zlib) [ 6813.075943] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 6813.292108] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6814.935183] Key type lgssc registered [ 6816.050260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6823.499887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6823.519611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6825.134770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6830.080401] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6835.696104] LustreError: 209134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.716909] LustreError: 209134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6838.920600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6839.406573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6844.947189] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6854.206265] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6861.011834] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6868.685898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6869.097372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6870.159834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6876.168065] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6886.042843] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6905.320580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6905.324935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6905.345237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6905.378746] Lustre: Skipped 1 previous similar message [ 6910.443052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6910.454611] Lustre: Skipped 2 previous similar messages [ 6910.915822] Lustre: server umount lustre-OST0000 complete [ 6915.559153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6915.574596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6915.581803] Lustre: Skipped 1 previous similar message [ 6915.594758] Lustre: Skipped 1 previous similar message [ 6920.673414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6920.683783] Lustre: Skipped 1 previous similar message [ 6925.792390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6925.801248] Lustre: Skipped 1 previous similar message [ 6929.375202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6929.705649] Lustre: server umount lustre-MDT0000 complete [ 6930.918639] LustreError: 209131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6930.933916] LustreError: 209131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6933.501864] LustreError: 209116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658059 with bad export cookie 11908953101290764675 [ 6933.505627] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.509149] LustreError: 209116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6933.851965] Lustre: server umount lustre-MDT0001 complete [ 6945.026346] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6948.477966] Key type lgssc unregistered [ 6948.761874] LNet: 212823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6948.767575] LNetError: 212823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6948.800775] LNet: Removed LNI 192.168.201.137@tcp [ 6949.827774] Key type .llcrypt unregistered [ 6949.830398] Key type ._llcrypt unregistered [ 6974.825945] Key type ._llcrypt registered [ 6974.828597] Key type .llcrypt registered [ 6974.947706] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6976.750312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6976.789984] alg: No test for adler32 (adler32-zlib) [ 6978.142177] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 6978.406813] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6980.159271] Key type lgssc registered [ 6981.563754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6991.309140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6991.346666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6992.989743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6998.648853] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7003.112830] LustreError: 214086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7003.125920] LustreError: 214086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7007.978417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7008.346931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7012.735868] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7020.677624] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7026.944487] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7034.578576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7034.906457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7038.018255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7042.371731] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7051.036535] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7068.642486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7068.651054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7068.673081] Lustre: Skipped 1 previous similar message [ 7068.675216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7071.376414] Lustre: server umount lustre-OST0000 complete [ 7078.882995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7078.897989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7078.900315] Lustre: Skipped 1 previous similar message [ 7081.637809] Lustre: server umount lustre-MDT0000 complete [ 7084.003409] LustreError: 214086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7084.018293] LustreError: 214086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7085.406340] LustreError: 214069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658211 with bad export cookie 6810588712550464846 [ 7085.420373] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7085.945911] Lustre: server umount lustre-MDT0001 complete [ 7099.380858] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7102.803714] Key type lgssc unregistered [ 7103.136913] LNet: 217769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7103.156532] LNetError: 217769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7103.175072] LNet: Removed LNI 192.168.201.137@tcp [ 7104.408676] Key type .llcrypt unregistered [ 7104.410714] Key type ._llcrypt unregistered [ 7124.186642] Key type ._llcrypt registered [ 7124.191260] Key type .llcrypt registered [ 7124.299139] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7125.212362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7125.277373] alg: No test for adler32 (adler32-zlib) [ 7126.394501] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 7126.714378] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7128.352163] Key type lgssc registered [ 7129.581190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7138.765618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7138.803494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7140.549936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7145.565192] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7150.051086] LustreError: 219033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7150.064793] LustreError: 219033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7154.981466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.356082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7160.533618] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7170.973498] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7178.565158] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7186.278824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7186.707654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7191.745985] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7201.179566] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7210.228026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7220.447668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7220.543684] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7220.786066] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7227.243625] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7234.695368] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7236.178274] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7248.126623] Lustre: Permanently deactivating lustre-OST0001 [ 7248.151542] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7257.723895] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7265.746850] Lustre: Permanently reactivating lustre-OST0001 [ 7265.764416] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7265.770122] Lustre: Skipped 2 previous similar messages [ 7274.634272] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.652345] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7274.662055] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7274.684431] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7274.737758] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7277.548871] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7277.551927] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7277.569156] Lustre: Skipped 2 previous similar messages [ 7277.582114] Lustre: Skipped 1 previous similar message [ 7278.209867] Lustre: server umount lustre-OST0001 complete [ 7282.660463] LustreError: 220855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7282.702221] LustreError: 220855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7287.266113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7287.268644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.294105] LustreError: Skipped 1 previous similar message [ 7287.295150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7287.328257] Lustre: Skipped 1 previous similar message [ 7287.781708] LustreError: 220858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7287.817743] LustreError: 220858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7288.783369] Lustre: server umount lustre-OST0000 complete [ 7292.897436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7292.918556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7292.928139] Lustre: Skipped 1 previous similar message [ 7292.941550] Lustre: Skipped 2 previous similar messages [ 7298.020974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7298.031218] Lustre: Skipped 1 previous similar message [ 7303.138444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7303.142882] Lustre: Skipped 1 previous similar message [ 7307.231160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7307.587163] Lustre: server umount lustre-MDT0000 complete [ 7308.265661] LustreError: 219029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7308.284938] LustreError: 219029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7311.738533] LustreError: 219015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658437 with bad export cookie 16986359875633369183 [ 7311.740655] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7311.749287] LustreError: 219015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7312.207432] Lustre: server umount lustre-MDT0001 complete [ 7323.083324] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7325.688588] Key type lgssc unregistered [ 7326.016992] LNet: 224124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7326.023921] LNetError: 224124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7326.036901] LNet: Removed LNI 192.168.201.137@tcp [ 7326.927111] Key type .llcrypt unregistered [ 7326.929847] Key type ._llcrypt unregistered [ 7369.915095] Key type ._llcrypt registered [ 7369.916702] Key type .llcrypt registered [ 7370.072992] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7371.459680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7371.658516] alg: No test for adler32 (adler32-zlib) [ 7372.734982] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 7372.999960] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7374.695669] Key type lgssc registered [ 7375.783523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7383.800901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7383.816486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.154791] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7385.176982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7385.333610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7389.593216] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7397.143228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7397.209372] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7397.226037] Lustre: 226329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7397.349593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7402.701290] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7412.167446] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7420.280331] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7428.213460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7428.345779] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7428.487602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7434.618966] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7434.812651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7442.111452] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7446.292516] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7452.088527] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7460.100457] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7468.072710] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7484.892118] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7484.903048] Lustre: Skipped 1 previous similar message [ 7507.617996] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7507.620852] Lustre: Skipped 2 previous similar messages [ 7542.300924] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7542.308660] Lustre: Skipped 3 previous similar messages [ 7607.912628] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7607.918784] Lustre: Skipped 8 previous similar messages [ 7619.047365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.058129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7619.060966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7619.062454] Lustre: Skipped 1 previous similar message [ 7620.848652] Lustre: server umount lustre-OST0000 complete [ 7627.745198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7627.753398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7627.765912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7627.771568] Lustre: Skipped 1 previous similar message [ 7629.281707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7629.285237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.702504] Lustre: server umount lustre-MDT0000 complete [ 7634.103755] LustreError: 226321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658760 with bad export cookie 13962824936432349367 [ 7634.106682] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7634.120799] LustreError: 226321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7634.696241] Lustre: server umount lustre-MDT0001 complete [ 7645.650339] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7648.124511] Key type lgssc unregistered [ 7648.440590] LNet: 231109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7648.444895] LNetError: 231109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7648.470731] LNet: Removed LNI 192.168.201.137@tcp [ 7649.238583] Key type .llcrypt unregistered [ 7649.240910] Key type ._llcrypt unregistered [ 7678.938848] Key type ._llcrypt registered [ 7678.941958] Key type .llcrypt registered [ 7679.061655] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7680.075731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7680.177286] alg: No test for adler32 (adler32-zlib) [ 7681.121505] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 7681.308382] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7683.127315] Key type lgssc registered [ 7684.219031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7691.786591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7691.805317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7693.240141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7697.746333] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7703.522879] LustreError: 232526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7703.546321] LustreError: 232526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7705.555518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7705.828830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7709.177300] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7717.703681] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7724.352084] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7732.611372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7732.983319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.193772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7741.522802] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7751.402795] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7772.136372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7772.141317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7772.163259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7777.119112] Lustre: server umount lustre-OST0000 complete [ 7782.372869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7782.380265] Lustre: Skipped 1 previous similar message [ 7782.386198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7782.389680] Lustre: Skipped 2 previous similar messages [ 7787.458772] Lustre: server umount lustre-MDT0000 complete [ 7787.500490] LustreError: 232521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7787.512351] LustreError: 232521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7791.553414] LustreError: 232506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658917 with bad export cookie 3615508175231872734 [ 7791.564591] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7791.898690] Lustre: server umount lustre-MDT0001 complete [ 7803.927991] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7806.910247] Key type lgssc unregistered [ 7807.169286] LNet: 236217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7807.179269] LNetError: 236217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7808.235632] LNet: Removed LNI 192.168.201.137@tcp [ 7809.230648] Key type .llcrypt unregistered [ 7809.233961] Key type ._llcrypt unregistered [ 7832.120412] Key type ._llcrypt registered [ 7832.123227] Key type .llcrypt registered [ 7832.225101] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7833.173747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7833.222626] alg: No test for adler32 (adler32-zlib) [ 7834.314662] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 7834.514972] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7836.151406] Key type lgssc registered [ 7837.140736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8108.774504] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 8111.313747] Key type lgssc unregistered [ 8111.737500] LNet: 238253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8111.743446] LNetError: 238253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8111.759694] LNet: Removed LNI 192.168.201.137@tcp [ 8112.792633] Key type .llcrypt unregistered [ 8112.797784] Key type ._llcrypt unregistered [ 8141.946776] Key type ._llcrypt registered [ 8141.949848] Key type .llcrypt registered [ 8142.093424] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 8158.990790] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8160.751204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8160.843943] alg: No test for adler32 (adler32-zlib) [ 8161.894434] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 8162.112678] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 8163.887193] Key type lgssc registered [ 8164.996338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8202.553964] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7886 sec ========== 21:48:47 (1788659327) [ 8204.330524] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:48:49 (1788659329) === [ 8207.981954] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:48:53 (1788659333) === [ 8256.046031] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 8260.147150] Key type lgssc unregistered [ 8260.533489] LNet: 243489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8260.540863] LNetError: 243489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8260.552529] LNet: Removed LNI 192.168.201.137@tcp [ 8261.937331] Key type .llcrypt unregistered [ 8261.938709] Key type ._llcrypt unregistered