[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 547021215 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003250] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008596] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011150] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.014086] *** VALIDATE selinux *** [ 0.022532] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027101] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028186] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030122] *** VALIDATE tmpfs *** [ 0.032303] *** VALIDATE proc *** [ 0.033271] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035302] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042450] debug: unmapping init [mem 0xffffffff8e259000-0xffffffff8e260fff] [ 0.044196] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045725] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053322] rcu: Hierarchical SRCU implementation. [ 0.055632] smp: Bringing up secondary CPUs ... [ 0.056606] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060555] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148336] node 0 deferred pages initialised in 81ms [ 0.151155] devtmpfs: initialized [ 0.153508] x86/mm: Memory block size: 128MB [ 0.157288] gcov: version magic: 0x41383552 [ 0.161452] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163129] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165427] pinctrl core: initialized pinctrl subsystem [ 0.168292] [ 0.170015] ************************************************************* [ 0.171013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174019] ** ** [ 0.176018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178022] ** ** [ 0.181023] ** This means that this kernel is built to expose internal ** [ 0.183017] ** IOMMU data structures, which may compromise security on ** [ 0.186023] ** your system. ** [ 0.188020] ** ** [ 0.192025] ** If you see this message and you are not debugging the ** [ 0.195019] ** kernel, report this immediately to your vendor! ** [ 0.197020] ** ** [ 0.200022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.202018] ************************************************************* [ 0.205920] NET: Registered protocol family 16 [ 0.207586] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.210099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.213102] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.217147] cpuidle: using governor menu [ 0.218985] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.221745] PCI: Using configuration type 1 for base access [ 0.225137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.234146] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.237067] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.241120] cryptd: max_cpu_qlen set to 1000 [ 0.244282] ACPI: Added _OSI(Module Device) [ 0.246012] ACPI: Added _OSI(Processor Device) [ 0.247010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.248010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.252573] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.259341] ACPI: Interpreter enabled [ 0.261068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.262015] ACPI: Using IOAPIC for interrupt routing [ 0.264106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.268267] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.280964] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.283069] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.286142] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.290120] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.296813] acpiphp: Slot [2] registered [ 0.298315] acpiphp: Slot [5] registered [ 0.300269] acpiphp: Slot [6] registered [ 0.302169] acpiphp: Slot [7] registered [ 0.303169] acpiphp: Slot [8] registered [ 0.305171] acpiphp: Slot [9] registered [ 0.306226] acpiphp: Slot [10] registered [ 0.307225] acpiphp: Slot [3] registered [ 0.309130] acpiphp: Slot [4] registered [ 0.311128] acpiphp: Slot [11] registered [ 0.312086] acpiphp: Slot [12] registered [ 0.314096] acpiphp: Slot [13] registered [ 0.315046] acpiphp: Slot [14] registered [ 0.316056] acpiphp: Slot [15] registered [ 0.317106] acpiphp: Slot [16] registered [ 0.318220] acpiphp: Slot [17] registered [ 0.320208] acpiphp: Slot [18] registered [ 0.322136] acpiphp: Slot [19] registered [ 0.323189] acpiphp: Slot [20] registered [ 0.325103] acpiphp: Slot [21] registered [ 0.327103] acpiphp: Slot [22] registered [ 0.328198] acpiphp: Slot [23] registered [ 0.330181] acpiphp: Slot [24] registered [ 0.332140] acpiphp: Slot [25] registered [ 0.333102] acpiphp: Slot [26] registered [ 0.335133] acpiphp: Slot [27] registered [ 0.337093] acpiphp: Slot [28] registered [ 0.338090] acpiphp: Slot [29] registered [ 0.339081] acpiphp: Slot [30] registered [ 0.341113] acpiphp: Slot [31] registered [ 0.342086] PCI host bridge to bus 0000:00 [ 0.344021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.346033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.348051] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.351036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.353032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.356035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.357216] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.361097] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.364630] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.373929] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.380038] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.383029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.386028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.388029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.392178] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.393565] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.396044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.401015] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.406015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.419017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.423018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.428368] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.436027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.444032] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458030] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.478508] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.490019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.505033] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.529018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.538804] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.548024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.555027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.573017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.581258] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.586018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.592020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.608023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.619431] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.627020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.633018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.658018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.674984] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.681022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.687022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.701035] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.712630] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.714345] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.716318] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.718356] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.720293] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.726098] iommu: Default domain type: Passthrough [ 0.728467] SCSI subsystem initialized [ 0.730223] ACPI: bus type USB registered [ 0.731112] usbcore: registered new interface driver usbfs [ 0.733121] usbcore: registered new interface driver hub [ 0.735124] usbcore: registered new device driver usb [ 0.738250] pps_core: LinuxPPS API ver. 1 registered [ 0.740015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.743128] PTP clock support registered [ 0.746350] EDAC MC: Ver: 3.0.0 [ 0.749156] PCI: Using ACPI for IRQ routing [ 0.750000] NetLabel: Initializing [ 0.750014] NetLabel: domain hash size = 128 [ 0.751000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.754113] NetLabel: unlabeled traffic allowed by default [ 0.756557] vgaarb: loaded [ 0.758795] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.761016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.770000] clocksource: Switched to clocksource kvm-clock [ 0.868509] VFS: Disk quotas dquot_6.6.0 [ 0.870602] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.873961] *** VALIDATE ramfs *** [ 0.875158] *** VALIDATE hugetlbfs *** [ 0.876649] pnp: PnP ACPI init [ 0.879094] pnp: PnP ACPI: found 6 devices [ 0.895228] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.897824] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.899564] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.901870] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.905883] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.907532] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909797] NET: Registered protocol family 2 [ 0.912170] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.917098] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.921942] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.928396] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.932056] TCP: Hash tables configured (established 65536 bind 65536) [ 0.935089] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.938456] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.941602] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.944616] NET: Registered protocol family 1 [ 0.948188] RPC: Registered named UNIX socket transport module. [ 0.950119] RPC: Registered udp transport module. [ 0.951785] RPC: Registered tcp transport module. [ 0.953889] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.956553] NET: Registered protocol family 44 [ 0.958656] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.961232] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.962935] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.964807] PCI: CLS 0 bytes, default 64 [ 0.966598] Unpacking initramfs... [ 2.351792] debug: unmapping init [mem 0xffff9d977cc54000-0xffff9d977ffbffff] [ 2.354409] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.356088] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.358955] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.862337] Initialise system trusted keyrings [ 2.864190] Key type blacklist registered [ 2.866507] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.880802] zbud: loaded [ 2.884770] *** VALIDATE nfs *** [ 2.886369] *** VALIDATE nfs4 *** [ 2.888359] pstore: using deflate compression [ 2.892172] Platform Keyring initialized [ 2.991110] NET: Registered protocol family 38 [ 2.992812] Key type asymmetric registered [ 2.994480] Asymmetric key parser 'x509' registered [ 2.996399] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.999520] io scheduler mq-deadline registered [ 3.001162] io scheduler kyber registered [ 3.004839] io scheduler bfq registered [ 3.007102] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.010538] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.013742] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.017240] ACPI: Power Button [PWRF] [ 3.024176] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.030979] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.043114] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.051368] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.066655] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.093849] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.124866] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.131153] Non-volatile memory driver v1.3 [ 3.132413] Linux agpgart interface v0.103 [ 3.172431] virtio_blk virtio1: [vda] 146000 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.176474] vda: detected capacity change from 0 to 74752000 [ 3.191371] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.193886] vdb: detected capacity change from 0 to 1073741824 [ 3.213048] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.218214] vdc: detected capacity change from 0 to 2621440000 [ 3.234241] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.236556] vdd: detected capacity change from 0 to 2621440000 [ 3.252131] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.254428] vde: detected capacity change from 0 to 4294967296 [ 3.273600] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.276317] vdf: detected capacity change from 0 to 4294967296 [ 3.285089] libphy: Fixed MDIO Bus: probed [ 3.298102] usbcore: registered new interface driver usbserial_generic [ 3.300526] usbserial: USB Serial support registered for generic [ 3.302861] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.307240] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.309030] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.311434] mousedev: PS/2 mouse device common for all mice [ 3.314381] rtc_cmos 00:05: RTC can wake from S4 [ 3.317034] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.320718] rtc_cmos 00:05: registered as rtc0 [ 3.323341] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.325412] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.327900] intel_pstate: CPU model not supported [ 3.330203] hid: raw HID events driver (C) Jiri Kosina [ 3.335383] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.338263] usbcore: registered new interface driver usbhid [ 3.338270] usbhid: USB HID core driver [ 3.338439] drop_monitor: Initializing network drop monitor service [ 3.338570] Initializing XFRM netlink socket [ 3.340067] NET: Registered protocol family 10 [ 3.354862] Segment Routing with IPv6 [ 3.356331] NET: Registered protocol family 17 [ 3.358357] mpls_gso: MPLS GSO support [ 3.362922] RAS: Correctable Errors collector initialized. [ 3.365251] AVX version of gcm_enc/dec engaged. [ 3.367498] AES CTR mode by8 optimization enabled [ 3.448081] sched_clock: Marking stable (3448027780, 0)->(4392206423, -944178643) [ 3.451176] registered taskstats version 1 [ 3.453052] Loading compiled-in X.509 certificates [ 3.454776] zswap: loaded using pool lzo/zbud [ 3.487392] Key type big_key registered [ 3.500657] Key type encrypted registered [ 3.502316] ima: No TPM chip found, activating TPM-bypass! [ 3.504270] ima: Allocated hash algorithm: sha1 [ 3.505846] ima: No architecture policies found [ 3.507563] evm: Initialising EVM extended attributes: [ 3.509430] evm: security.selinux [ 3.511846] evm: security.ima [ 3.512876] evm: security.capability [ 3.514259] evm: HMAC attrs: 0x1 [ 3.517188] rtc_cmos 00:05: setting system clock to 2026-08-19 04:49:35 UTC (1787114975) [ 3.523851] debug: unmapping init [mem 0xffffffff8f203000-0xffffffff8f3fffff] [ 3.527317] debug: unmapping init [mem 0xffffffff8df82000-0xffffffff8e258fff] [ 3.535343] Write protecting the kernel read-only data: 28672k [ 3.538814] debug: unmapping init [mem 0xffffffff8c603000-0xffffffff8c7fffff] [ 3.542325] debug: unmapping init [mem 0xffffffff8cf14000-0xffffffff8cffffff] [ 3.590468] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.599557] systemd[1]: Detected virtualization kvm. [ 3.601533] systemd[1]: Detected architecture x86-64. [ 3.604307] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.634983] systemd[1]: No hostname configured. [ 3.637159] systemd[1]: Set hostname to . [ 3.639654] random: systemd: uninitialized urandom read (16 bytes read) [ 3.642687] systemd[1]: Initializing machine ID from random generator. [ 3.692187] random: ln: uninitialized urandom read (6 bytes read) [ 3.810367] random: systemd: uninitialized urandom read (16 bytes read) [ 3.813457] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.818080] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.822898] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.454527] device-mapper: uevent: version 1.0.3 [ 4.456626] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK [0[ 5.223060] virtio_net virtio0 ens2: renamed from eth0 m] Reached target Basic System. [[ 5.264994] random: fast init done  OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.304963] scsi host0: ata_piix [ 5.322507] scsi host1: ata_piix [ 5.324375] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.326734] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.917707] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.020698] random: crng init done [ 10.022061] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.366518] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.546133] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.822867] SELinux: Disabled at runtime. [ 11.887910] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.896592] systemd[1]: Detected virtualization kvm. [ 11.898604] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.404765] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.408664] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.414368] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.419308] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.422405] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.441499] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.452129] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ 12.475326] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.041268] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.370428] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.394794] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.516346] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.541548] EDAC sbridge: Ver: 1.1.2 [ 15.193954] Key type dns_resolver registered [ 15.510461] NFS: Registering the id_resolver key type [ 15.512734] Key type id_resolver registered [ 15.514457] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg202-server login: [ 44.426649] libcfs: loading out-of-tree module taints kernel. [ 44.515658] Key type ._llcrypt registered [ 44.518588] Key type .llcrypt registered [ 44.618926] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 66.713558] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 68.534740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.572393] alg: No test for adler32 (adler32-zlib) [ 70.409428] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 71.820872] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 73.623790] Key type lgssc registered [ 76.897540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.718940] hrtimer: interrupt took 4245920 ns [ 99.383707] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 144.747534] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 159.563465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.578029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.031234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 161.080278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 161.172797] Lustre: lustre-MDT0000: new disk, initializing [ 161.241875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 161.259556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.656690] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.353116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.513801] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 180.575703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 180.586388] Lustre: Skipped 1 previous similar message [ 180.747138] Lustre: lustre-MDT0001: new disk, initializing [ 180.879104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.920348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 180.946611] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.899634] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.006650] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.847232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.347951] Lustre: lustre-OST0000: new disk, initializing [ 205.357959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 205.380362] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.551520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.014716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.038878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.272142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 213.947821] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.215982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.369116] Lustre: lustre-OST0001: new disk, initializing [ 231.373947] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 231.383398] Lustre: 9521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.458732] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.609499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 237.619160] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 237.718437] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 238.896959] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.748918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 263.293484] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 271.971363] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing check_logdir /tmp/testlogs/ [ 277.122590] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing yml_node [ 281.816070] Lustre: DEBUG MARKER: Client: 2.17.56.3 [ 284.656872] Lustre: DEBUG MARKER: MDS: 2.17.56.3 [ 287.478265] Lustre: DEBUG MARKER: OSS: 2.17.56.3 [ 288.803805] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 00:54:18 EDT 2026 [ 305.519438] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 306.874900] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 324.583623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 324.593936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.616145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.208968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.214853] Lustre: Skipped 7 previous similar messages [ 335.333795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.345209] Lustre: Skipped 3 previous similar messages [ 337.375169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 339.619542] Lustre: server umount lustre-MDT0000 complete [ 343.467877] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115315 with bad export cookie 1827205407603614446 [ 343.486536] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.569161] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.570429] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.574146] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 345.594860] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 345.610050] Lustre: Skipped 4 previous similar messages [ 348.576157] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.604405] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 349.793647] Lustre: server umount lustre-MDT0001 complete [ 354.093601] Lustre: server umount lustre-OST0000 complete [ 358.561481] Lustre: server umount lustre-OST0001 complete [ 366.698739] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 379.055614] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 416.436201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.659860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 416.705592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.816072] Lustre: lustre-MDT0000: new disk, initializing [ 416.917551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 417.012580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.292728] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 431.820043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.881293] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 431.926785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 431.930227] Lustre: Skipped 1 previous similar message [ 431.994912] Lustre: lustre-MDT0001: new disk, initializing [ 432.082220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 432.118171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 432.128741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 436.185852] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 445.846423] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 452.653179] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 460.293631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 460.633527] Lustre: lustre-OST0000: new disk, initializing [ 460.639122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 460.653127] Lustre: 17048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 460.752354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.534329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 462.544296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 462.652830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 466.997387] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 476.701441] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 480.787487] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 481.012729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 485.886172] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 486.105646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 488.426308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.430064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 488.433109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 488.433117] Lustre: Skipped 3 previous similar messages [ 488.442947] Lustre: Skipped 1 previous similar message [ 498.658471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.666271] Lustre: Skipped 4 previous similar messages [ 502.240085] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 502.449885] Lustre: server umount lustre-OST0000 complete [ 508.903645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.914921] Lustre: Skipped 1 previous similar message [ 508.930088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 512.393960] Lustre: server umount lustre-MDT0000 complete [ 514.027991] LustreError: 15961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.043569] LustreError: 15961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 516.252096] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115488 with bad export cookie 1827205407603615979 [ 516.269724] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.276546] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 516.539106] Lustre: server umount lustre-MDT0001 complete [ 519.785893] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 00:58:09 (1787115489) [ 527.365546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.820836] LustreError: 19315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.953695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.393688] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 532.971806] LustreError: 19316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.938515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.018450] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 554.503527] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 562.173927] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 569.245551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 575.950785] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 583.197392] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 593.378253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 593.389430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.408258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 593.420022] Lustre: Skipped 2 previous similar messages [ 597.472929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 599.449961] Lustre: server umount lustre-OST0000 complete [ 603.115248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.127127] Lustre: Skipped 1 previous similar message [ 609.234769] Lustre: server umount lustre-MDT0000 complete [ 612.702951] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115584 with bad export cookie 1827205407603616861 [ 612.710792] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.017929] Lustre: server umount lustre-MDT0001 complete [ 623.989367] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 627.669665] Key type lgssc unregistered [ 627.958029] LNet: 22848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.965454] LNetError: 22848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.983764] LNet: Removed LNI 192.168.202.102@tcp [ 629.013414] Key type .llcrypt unregistered [ 629.014946] Key type ._llcrypt unregistered [ 653.759214] Key type ._llcrypt registered [ 653.761177] Key type .llcrypt registered [ 653.840634] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 654.827205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 654.854577] alg: No test for adler32 (adler32-zlib) [ 655.893417] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 656.168553] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 657.784095] Key type lgssc registered [ 658.690326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 665.310701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 665.340430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.754820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 671.001587] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 676.329493] LustreError: 24267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 676.343377] LustreError: 24267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 678.905345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.213301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 683.059365] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 689.487426] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 695.965563] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 703.814762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 704.362926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 710.648183] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 711.572417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 717.323733] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 732.140566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 732.141021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 732.157183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 732.170090] Lustre: Skipped 1 previous similar message [ 733.543233] Lustre: server umount lustre-OST0000 complete [ 737.248666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.252204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 737.262820] Lustre: Skipped 1 previous similar message [ 737.274908] Lustre: Skipped 2 previous similar messages [ 742.369649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.374752] Lustre: Skipped 1 previous similar message [ 743.049484] Lustre: server umount lustre-MDT0000 complete [ 747.071644] LustreError: 24248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115719 with bad export cookie 5998148429691481455 [ 747.072376] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 747.078759] LustreError: 24248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 747.337375] Lustre: server umount lustre-MDT0001 complete [ 758.992685] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 761.618995] Key type lgssc unregistered [ 761.878506] LNet: 28256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.895988] LNetError: 28256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.934613] LNet: Removed LNI 192.168.202.102@tcp [ 762.980066] Key type .llcrypt unregistered [ 762.984427] Key type ._llcrypt unregistered [ 787.678927] Key type ._llcrypt registered [ 787.682394] Key type .llcrypt registered [ 787.790729] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 789.177034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.197527] alg: No test for adler32 (adler32-zlib) [ 790.310328] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 790.640562] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 792.343175] Key type lgssc registered [ 793.835353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 802.193202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 802.232735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 804.142975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 809.995041] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 814.564961] LustreError: 29669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 814.578143] LustreError: 29669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 819.681954] LustreError: 29670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 820.308520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 820.763944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 825.914520] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 834.145805] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 839.445918] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 856.998344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 857.543127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 864.542519] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 872.117359] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 874.952970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 887.265395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 887.273048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.289584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 888.529287] Lustre: server umount lustre-OST0000 complete [ 895.460706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 895.468506] Lustre: Skipped 1 previous similar message [ 895.472746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.478355] Lustre: Skipped 1 previous similar message [ 900.581460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 900.592450] Lustre: Skipped 1 previous similar message [ 902.789179] Lustre: server umount lustre-MDT0000 complete [ 905.702741] LustreError: 29670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.718024] LustreError: 29670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 906.919615] LustreError: 29649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115878 with bad export cookie 17461664515392854634 [ 906.927386] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 906.940292] LustreError: 29649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 907.354852] Lustre: server umount lustre-MDT0001 complete [ 918.838371] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 921.794610] Key type lgssc unregistered [ 922.103681] LNet: 33575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.118259] LNetError: 33575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.144178] LNet: Removed LNI 192.168.202.102@tcp [ 923.143225] Key type .llcrypt unregistered [ 923.146235] Key type ._llcrypt unregistered [ 948.307667] Key type ._llcrypt registered [ 948.309993] Key type .llcrypt registered [ 948.519736] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 949.934149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 950.192110] alg: No test for adler32 (adler32-zlib) [ 951.309623] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 951.657764] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 953.431793] Key type lgssc registered [ 954.770511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 963.703809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 963.740363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 965.231505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 970.653503] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 975.841129] LustreError: 34991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.862684] LustreError: 34991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 979.039933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.594574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 983.868926] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 991.886122] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 998.720658] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1006.460450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1006.830295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.943063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1014.353190] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1023.024690] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1034.722369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1034.728825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1034.742064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1034.762394] Lustre: Skipped 1 previous similar message [ 1039.029262] Lustre: server umount lustre-OST0000 complete [ 1044.964423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.986404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1044.989088] Lustre: Skipped 2 previous similar messages [ 1049.078256] Lustre: server umount lustre-MDT0000 complete [ 1050.080255] LustreError: 34991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1050.098927] LustreError: 34991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1053.239082] LustreError: 34970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116025 with bad export cookie 5427817810311523677 [ 1053.259252] LustreError: 34970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1053.260805] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1053.594893] Lustre: server umount lustre-MDT0001 complete [ 1064.893708] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1067.475761] Key type lgssc unregistered [ 1067.794553] LNet: 38526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.797812] LNetError: 38526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.838900] LNet: Removed LNI 192.168.202.102@tcp [ 1069.739493] Key type .llcrypt unregistered [ 1069.745413] Key type ._llcrypt unregistered [ 1093.290980] Key type ._llcrypt registered [ 1093.295382] Key type .llcrypt registered [ 1093.397462] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1094.357284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1094.439644] alg: No test for adler32 (adler32-zlib) [ 1095.444253] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1095.647136] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1097.295170] Key type lgssc registered [ 1098.462251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1107.341695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1107.366949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.916781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.683846] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1118.180466] LustreError: 39940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.209824] LustreError: 39940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1123.304942] LustreError: 39941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1123.569587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1124.041910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1129.392202] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1139.322118] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1145.156808] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1152.602908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1153.028931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.057629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1159.383312] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1167.347856] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1174.501425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1174.518184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1174.558029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1178.595589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1178.602572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1178.604708] Lustre: Skipped 1 previous similar message [ 1178.613192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.643090] Lustre: lustre-OST0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1179.657263] Lustre: Skipped 1 previous similar message [ 1179.977626] Lustre: server umount lustre-OST0000 complete [ 1201.121913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1201.139592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.159640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1203.168845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.236761] Lustre: server umount lustre-MDT0000 complete [ 1208.288093] LustreError: 39941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.301564] LustreError: 39941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1210.118402] LustreError: 39922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116182 with bad export cookie 16724417269231032418 [ 1210.133979] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1216.451380] Lustre: server umount lustre-MDT0001 complete [ 1226.969662] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1229.562623] Key type lgssc unregistered [ 1229.812227] LNet: 43649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.822958] LNetError: 43649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.834712] LNet: Removed LNI 192.168.202.102@tcp [ 1230.559524] Key type .llcrypt unregistered [ 1230.561639] Key type ._llcrypt unregistered [ 1252.553299] Key type ._llcrypt registered [ 1252.555527] Key type .llcrypt registered [ 1252.664667] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1253.862134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1253.881780] alg: No test for adler32 (adler32-zlib) [ 1254.919767] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1255.127526] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1256.832282] Key type lgssc registered [ 1258.175418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1265.740289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1265.752079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.199553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.507397] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1277.417862] LustreError: 45061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.444739] LustreError: 45061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1279.060812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.361287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1283.632480] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1289.892678] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1294.762440] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1301.329203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1301.733849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1308.253640] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1313.040557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1316.914696] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1325.087988] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1325.541713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1325.553348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.611178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1328.614585] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.621570] Lustre: Skipped 2 previous similar messages [ 1328.634115] Lustre: Skipped 1 previous similar message [ 1330.158782] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1332.692838] Lustre: server umount lustre-MDT0000 complete [ 1333.749601] LustreError: 45062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1333.775549] LustreError: 45062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1335.289664] LustreError: 47612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.215330] LustreError: 45042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116308 with bad export cookie 13535572095978185755 [ 1336.226977] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1338.850588] LustreError: 45058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.852200] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.879608] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1342.677643] Lustre: server umount lustre-MDT0001 complete [ 1363.978133] Lustre: server umount lustre-OST0000 complete [ 1381.535235] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1384.126289] Key type lgssc unregistered [ 1384.352600] LNet: 48925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.355597] LNetError: 48925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1384.370879] LNet: Removed LNI 192.168.202.102@tcp [ 1385.055643] Key type .llcrypt unregistered [ 1385.057284] Key type ._llcrypt unregistered [ 1408.833438] Key type ._llcrypt registered [ 1408.835881] Key type .llcrypt registered [ 1408.998828] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1410.384046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1410.409485] alg: No test for adler32 (adler32-zlib) [ 1411.466207] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1411.804905] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1413.559203] Key type lgssc registered [ 1414.675342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1422.001946] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1422.021114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1437.536111] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1442.656911] LustreError: 50321:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 1469.343401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.688443] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1482.925836] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1647.895618] Lustre: server umount lustre-OST0000 complete [ 1665.489293] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1667.846548] Key type lgssc unregistered [ 1668.062413] LNet: 51996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.067870] LNetError: 51996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.079791] LNet: Removed LNI 192.168.202.102@tcp [ 1668.950455] Key type .llcrypt unregistered [ 1668.952264] Key type ._llcrypt unregistered [ 1692.349922] Key type ._llcrypt registered [ 1692.353852] Key type .llcrypt registered [ 1692.533805] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1693.709893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1693.778459] alg: No test for adler32 (adler32-zlib) [ 1694.901810] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1695.198535] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1696.919132] Key type lgssc registered [ 1697.873591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1705.066677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1705.087654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.544264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1710.756121] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1716.704396] LustreError: 53414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.728280] LustreError: 53414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1718.728609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.949551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1723.045520] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1730.581218] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1737.075644] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1743.642740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1743.851252] LustreError: 55232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1744.048259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.326219] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1751.553151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1759.378400] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1764.834185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1764.841655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.860913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1766.892685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1766.896280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1766.902949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1768.334328] Lustre: server umount lustre-OST0000 complete [ 1775.586129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1775.593413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.620987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.629304] Lustre: Skipped 1 previous similar message [ 1778.895567] Lustre: server umount lustre-MDT0000 complete [ 1782.241498] LustreError: 53413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1782.274373] LustreError: 53413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1783.154032] LustreError: 53393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116755 with bad export cookie 1169968513646395793 [ 1783.170303] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1783.589428] Lustre: server umount lustre-MDT0001 complete [ 1793.829335] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1796.166711] Key type lgssc unregistered [ 1796.342679] LNet: 56942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.347917] LNetError: 56942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1796.362521] LNet: Removed LNI 192.168.202.102@tcp [ 1797.035988] Key type .llcrypt unregistered [ 1797.037637] Key type ._llcrypt unregistered [ 1818.194334] Key type ._llcrypt registered [ 1818.195950] Key type .llcrypt registered [ 1818.286596] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1819.005960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.040514] alg: No test for adler32 (adler32-zlib) [ 1820.346871] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1820.574571] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1822.225217] Key type lgssc registered [ 1823.451770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1830.516144] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1830.561141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1846.112918] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1851.231417] LustreError: 58321:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 1877.983271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.667631] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1891.175378] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1897.535156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.977343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.051579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1902.301544] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1903.072546] LustreError: 58346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1903.119355] LustreError: 58346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1908.197035] LustreError: 58347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1911.310043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.691251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1916.128901] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1924.324357] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1930.506232] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1933.290152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.294131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1933.304151] Lustre: Skipped 1 previous similar message [ 1933.317457] Lustre: Skipped 1 previous similar message [ 1938.405854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1938.420332] Lustre: Skipped 1 previous similar message [ 1938.653726] Lustre: server umount lustre-OST0000 complete [ 1964.003064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.005351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.027134] Lustre: Skipped 1 previous similar message [ 1964.043885] Lustre: Skipped 1 previous similar message [ 1970.128917] Lustre: server umount lustre-MDT0000 complete [ 1970.154187] LustreError: 59361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1970.179787] LustreError: 59361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1974.713771] LustreError: 58327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116946 with bad export cookie 8569411713272413944 [ 1974.726485] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.729173] LustreError: 58327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1989.093880] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1989.484675] Lustre: server umount lustre-MDT0001 complete [ 2001.177115] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2004.450514] Key type lgssc unregistered [ 2004.878700] LNet: 62067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2004.894471] LNetError: 62067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2004.912884] LNet: Removed LNI 192.168.202.102@tcp [ 2005.971777] Key type .llcrypt unregistered [ 2005.974434] Key type ._llcrypt unregistered [ 2031.200869] Key type ._llcrypt registered [ 2031.202645] Key type .llcrypt registered [ 2031.309308] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2032.497640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2032.544308] alg: No test for adler32 (adler32-zlib) [ 2033.749995] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2034.150198] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2035.891712] Key type lgssc registered [ 2037.323263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2046.130075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2046.155984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.952436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2053.543599] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2057.192798] LustreError: 63482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2057.214763] LustreError: 63482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2062.307631] LustreError: 63481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.751555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.271191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2069.093527] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2076.571050] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2082.150687] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2088.968837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2089.425734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2092.631711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2096.753455] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2105.061670] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2123.232416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2123.235859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.253087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2125.221534] Lustre: server umount lustre-OST0000 complete [ 2130.912596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2130.919312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.944061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.972530] Lustre: server umount lustre-MDT0000 complete [ 2136.552794] LustreError: 64485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.568917] LustreError: 64485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2139.078601] LustreError: 66284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117111 with bad export cookie 14349284833068664355 [ 2139.079645] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.088544] LustreError: 66284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2149.533590] Lustre: server umount lustre-MDT0001 complete [ 2160.239162] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2163.132834] Key type lgssc unregistered [ 2163.474994] LNet: 67037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2163.487455] LNetError: 67037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2163.510370] LNet: Removed LNI 192.168.202.102@tcp [ 2164.458498] Key type .llcrypt unregistered [ 2164.462523] Key type ._llcrypt unregistered [ 2191.411910] Key type ._llcrypt registered [ 2191.414400] Key type .llcrypt registered [ 2191.515455] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2192.915189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2192.993724] alg: No test for adler32 (adler32-zlib) [ 2194.118302] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2194.550250] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2196.223748] Key type lgssc registered [ 2197.598788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2206.487937] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 01:26:15 (1787117175) [ 2213.500726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2213.527331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.936351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2219.717543] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2224.101413] LustreError: 68743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.128564] LustreError: 68743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2228.181769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.655471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2233.176054] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2239.673132] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2245.019755] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2252.351514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2252.791849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.644186] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2259.971033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2265.128836] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2270.467482] Lustre: Failing over lustre-MDT0000 [ 2270.796773] Lustre: server umount lustre-MDT0000 complete [ 2271.749743] LustreError: 70215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.786570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2274.793910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.804543] LustreError: 68744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.307887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2276.862637] LustreError: 70215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.885800] LustreError: 70215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2279.644075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.717100] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.822696] Lustre: *** cfs_fail_loc=135, val=0*** [ 2279.895761] LustreError: 71831:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2279.903398] LustreError: 71831:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.102@tcp: cfg command failed: rc = -2 [ 2279.912324] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2279.912324] [ 2279.932435] LustreError: MGC192.168.202.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2279.949684] LustreError: 71819:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2279.971768] LustreError: 71819:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2279.987547] LustreError: 71819:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2279.991230] LustreError: 71819:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2280.014748] Lustre: server umount lustre-MDT0000 complete [ 2280.016582] LustreError: 71819:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2281.964767] LustreError: 68739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.981552] LustreError: 68739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2285.967459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.066501] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.074910] LustreError: 72176:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.102@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000055177311) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2286.075112] LustreError: 68735:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2286.262350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.322336] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2287.094872] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2291.707932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2291.731577] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2291.798059] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2291.870136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2302.436721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2302.443717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.456690] Lustre: Skipped 1 previous similar message [ 2302.481266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2304.992405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2305.004762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.035409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2307.042611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2308.823227] Lustre: server umount lustre-OST0000 complete [ 2317.282126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.308785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2317.311287] Lustre: Skipped 1 previous similar message [ 2322.401873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2322.410996] Lustre: Skipped 1 previous similar message [ 2328.031946] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2330.296237] Lustre: server umount lustre-MDT0000 complete [ 2332.642211] LustreError: 68744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.672792] LustreError: 68744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2334.382632] LustreError: 68722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117306 with bad export cookie 5302153915393929330 [ 2334.401133] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2334.411281] LustreError: 68722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2334.730049] Lustre: server umount lustre-MDT0001 complete [ 2347.505957] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2350.737508] Key type lgssc unregistered [ 2351.028435] LNet: 73614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.031743] LNetError: 73614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.045437] LNet: Removed LNI 192.168.202.102@tcp [ 2351.819322] Key type .llcrypt unregistered [ 2351.823219] Key type ._llcrypt unregistered [ 2372.644405] Key type ._llcrypt registered [ 2372.646477] Key type .llcrypt registered [ 2372.699961] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2373.903599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2373.933027] alg: No test for adler32 (adler32-zlib) [ 2374.979410] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2375.204555] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2376.903268] Key type lgssc registered [ 2377.967926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2385.829581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2385.850545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.483871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.497323] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2397.670151] LustreError: 75007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.689907] LustreError: 75007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2400.580474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.813047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2405.831521] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2415.389477] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2421.499503] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2428.350112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2428.723701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2433.838369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2434.529523] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2441.438608] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2452.122979] Lustre: Failing over lustre-MDT0000 [ 2452.700940] Lustre: server umount lustre-MDT0000 complete [ 2454.564650] LustreError: 75003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2455.524281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.546040] Lustre: Skipped 1 previous similar message [ 2459.646550] LustreError: 75003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.681112] LustreError: 75003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2464.694954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.751210] LustreError: 75002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.776398] LustreError: 75002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2464.847730] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2464.937977] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2465.100439] LustreError: 78098:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2465.113236] LustreError: 78098:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.102@tcp: cfg command failed: rc = -9 [ 2465.123405] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2465.123405] [ 2465.129319] LustreError: MGC192.168.202.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2465.141047] LustreError: 78085:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2465.157764] LustreError: 78085:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2465.168130] LustreError: 78085:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2465.182838] LustreError: 78085:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2465.224880] Lustre: server umount lustre-MDT0000 complete [ 2465.228281] LustreError: 78085:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2469.872088] LustreError: 75004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.895369] LustreError: 75004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2471.992652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2472.241972] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2472.248156] LustreError: 78444:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.102@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000c1d2cd9f) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2472.264523] LustreError: 74999:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2472.569090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2472.629958] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2474.808858] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2477.220842] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2478.066338] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2478.118265] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2478.196092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2488.292618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.298775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2488.314525] Lustre: Skipped 2 previous similar messages [ 2488.324467] Lustre: Skipped 1 previous similar message [ 2493.427328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2493.439457] Lustre: Skipped 1 previous similar message [ 2493.633567] Lustre: server umount lustre-OST0000 complete [ 2498.545720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2498.555449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.561827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.578890] Lustre: Skipped 1 previous similar message [ 2503.654223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2503.659810] Lustre: Skipped 1 previous similar message [ 2503.668996] LustreError: 75004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.680611] LustreError: 75004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2503.716480] Lustre: server umount lustre-MDT0000 complete [ 2507.629599] LustreError: 74987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117479 with bad export cookie 13714264198783536596 [ 2507.635195] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2507.643843] LustreError: 74987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2508.064394] Lustre: server umount lustre-MDT0001 complete [ 2521.099811] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2523.475528] Key type lgssc unregistered [ 2523.776870] LNet: 79877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.787862] LNetError: 79877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2523.814169] LNet: Removed LNI 192.168.202.102@tcp [ 2524.627873] Key type .llcrypt unregistered [ 2524.632296] Key type ._llcrypt unregistered [ 2537.408960] Key type ._llcrypt registered [ 2537.411671] Key type .llcrypt registered [ 2537.506405] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2537.738344] Key type .llcrypt unregistered [ 2537.740401] Key type ._llcrypt unregistered [ 2563.400964] Key type ._llcrypt registered [ 2563.405781] Key type .llcrypt registered [ 2563.504662] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2565.014153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2565.072146] alg: No test for adler32 (adler32-zlib) [ 2566.181618] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2566.454704] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2568.127191] Key type lgssc registered [ 2568.997623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2576.954309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2576.982175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.303961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.996522] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2588.641754] LustreError: 81778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.658513] LustreError: 81778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2590.053097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.478846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2595.223980] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2604.037880] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2610.563116] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2618.313290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2618.796976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.940614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2624.691119] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2632.093125] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2645.488216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2645.495486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2645.514796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2646.903246] Lustre: server umount lustre-OST0000 complete [ 2650.592812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.613225] Lustre: Skipped 2 previous similar messages [ 2650.615955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2650.625183] Lustre: Skipped 2 previous similar messages [ 2655.716947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2655.732857] Lustre: Skipped 1 previous similar message [ 2656.950283] Lustre: server umount lustre-MDT0000 complete [ 2660.832960] LustreError: 82505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.864221] LustreError: 82505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2662.384972] LustreError: 81760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117634 with bad export cookie 11164677033001787203 [ 2662.386405] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.430302] LustreError: 81760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2662.940488] Lustre: server umount lustre-MDT0001 complete [ 2677.289650] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2681.347117] Key type lgssc unregistered [ 2681.698090] LNet: 85338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2681.708628] LNetError: 85338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2681.723852] LNet: Removed LNI 192.168.202.102@tcp [ 2682.613175] Key type .llcrypt unregistered [ 2682.614934] Key type ._llcrypt unregistered [ 2707.785763] Key type ._llcrypt registered [ 2707.790748] Key type .llcrypt registered [ 2707.901959] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2708.811560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2708.931840] alg: No test for adler32 (adler32-zlib) [ 2710.069385] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2710.296730] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2711.983262] Key type lgssc registered [ 2713.385959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2721.349660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2721.364682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.212496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2728.189956] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2733.541608] LustreError: 86753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.563263] LustreError: 86753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2736.964604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.538153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2742.592532] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2751.752335] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2758.375152] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2766.935778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2767.256358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.318332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2774.907326] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2786.529229] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2792.931928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.934342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2792.949455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2798.061174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2798.069817] Lustre: Skipped 1 previous similar message [ 2798.768175] Lustre: server umount lustre-OST0000 complete [ 2803.172675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2803.198236] Lustre: Skipped 1 previous similar message [ 2803.201962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2803.206731] Lustre: Skipped 2 previous similar messages [ 2808.291336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2808.317068] Lustre: Skipped 1 previous similar message [ 2809.232297] Lustre: server umount lustre-MDT0000 complete [ 2813.423795] LustreError: 87481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.539606] LustreError: 86733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117786 with bad export cookie 2786411438105090344 [ 2814.544391] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2814.551625] LustreError: 86733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2815.450357] Lustre: server umount lustre-MDT0001 complete [ 2829.809444] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2833.041272] Key type lgssc unregistered [ 2833.330084] LNet: 90286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2833.346581] LNetError: 90286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2834.412441] LNet: Removed LNI 192.168.202.102@tcp [ 2835.437586] Key type .llcrypt unregistered [ 2835.440066] Key type ._llcrypt unregistered [ 2859.448570] Key type ._llcrypt registered [ 2859.453372] Key type .llcrypt registered [ 2859.505447] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2861.021991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2861.093285] alg: No test for adler32 (adler32-zlib) [ 2862.214922] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2862.544353] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2864.311181] Key type lgssc registered [ 2865.631763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2873.752405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2873.773804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.277250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2880.204300] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2885.614359] LustreError: 91699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.668019] LustreError: 91699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2889.145835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.679738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2894.066500] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2901.535385] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2907.581766] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2914.608420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2915.079987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2920.198650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2921.241191] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2929.439527] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2950.114224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2950.121365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.140293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.147912] Lustre: Skipped 1 previous similar message [ 2954.749287] Lustre: server umount lustre-OST0000 complete [ 2960.359023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.362659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.378811] Lustre: Skipped 2 previous similar messages [ 2960.402768] Lustre: Skipped 1 previous similar message [ 2965.476887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2965.497620] Lustre: Skipped 1 previous similar message [ 2970.602980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.608332] Lustre: Skipped 1 previous similar message [ 2973.151856] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2973.470941] Lustre: server umount lustre-MDT0000 complete [ 2975.718689] LustreError: 93132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.740787] LustreError: 93132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2977.758935] LustreError: 91681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117949 with bad export cookie 18238535972067616093 [ 2977.773764] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2978.455704] Lustre: server umount lustre-MDT0001 complete [ 2990.449843] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2993.072372] Key type lgssc unregistered [ 2993.322367] LNet: 95259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2993.332194] LNetError: 95259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2994.413419] LNet: Removed LNI 192.168.202.102@tcp [ 2995.069407] Key type .llcrypt unregistered [ 2995.071810] Key type ._llcrypt unregistered [ 3017.865715] Key type ._llcrypt registered [ 3017.868707] Key type .llcrypt registered [ 3018.012724] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3019.251796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3019.322507] alg: No test for adler32 (adler32-zlib) [ 3020.401317] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 3020.679025] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3022.343253] Key type lgssc registered [ 3023.510426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3031.141165] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3031.196556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3046.879383] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3051.999337] LustreError: 96650:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 3078.687476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.169209] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3094.145881] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3102.376505] Lustre: server umount lustre-OST0000 complete [ 3113.648532] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 01:41:22 (1787118082) [ 3121.942163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.474891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3127.782782] LustreError: 98290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.797522] LustreError: 98290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3127.803299] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3132.900339] LustreError: 98291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.540168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.849672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3141.865905] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3150.443578] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3156.120345] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3162.827456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3163.096052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.239878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3169.150742] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3176.582507] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3192.800576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3192.814965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.838565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3196.902522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3196.912814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3196.921651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3198.449236] Lustre: server umount lustre-OST0000 complete [ 3203.552796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3203.557957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.585438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3203.588049] Lustre: Skipped 1 previous similar message [ 3207.143952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3207.146531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3212.266160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3212.272497] Lustre: Skipped 2 previous similar messages [ 3217.379677] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3217.645972] Lustre: server umount lustre-MDT0000 complete [ 3222.443665] LustreError: 98270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118194 with bad export cookie 11911816560325758458 [ 3222.457878] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3222.725100] Lustre: server umount lustre-MDT0001 complete [ 3237.828232] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3241.388338] Key type lgssc unregistered [ 3241.739589] LNet: 101984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3241.747613] LNetError: 101984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3242.793567] LNet: Removed LNI 192.168.202.102@tcp [ 3243.737608] Key type .llcrypt unregistered [ 3243.739779] Key type ._llcrypt unregistered [ 3286.327271] Key type ._llcrypt registered [ 3286.329888] Key type .llcrypt registered [ 3286.403749] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3288.001102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3288.028412] alg: No test for adler32 (adler32-zlib) [ 3289.085285] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 3289.367487] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3291.015352] Key type lgssc registered [ 3292.019695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3299.165651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3299.197441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3300.631384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3305.990965] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3310.049211] LustreError: 104170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3310.064698] LustreError: 104170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3314.029781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3314.508284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3319.130138] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3327.725545] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3334.115337] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3340.977934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3341.282553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.442651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3348.679192] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3356.612540] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3368.935777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3368.947107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.970930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3369.952100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3369.956174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3369.961185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3370.017037] Lustre: Skipped 1 previous similar message [ 3373.804283] Lustre: server umount lustre-OST0000 complete [ 3380.198669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3380.208136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3380.212947] Lustre: Skipped 1 previous similar message [ 3380.226729] Lustre: Skipped 1 previous similar message [ 3384.472853] Lustre: server umount lustre-MDT0000 complete [ 3385.320196] LustreError: 106665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.344787] LustreError: 106665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3388.988599] LustreError: 104151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118360 with bad export cookie 363461599075074408 [ 3388.994580] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3389.002842] LustreError: 104151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3389.492392] Lustre: server umount lustre-MDT0001 complete [ 3404.658745] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3409.831502] Key type lgssc unregistered [ 3410.213415] LNet: 107707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3410.232768] LNetError: 107707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3410.261791] LNet: Removed LNI 192.168.202.102@tcp [ 3411.934165] Key type .llcrypt unregistered [ 3411.941942] Key type ._llcrypt unregistered [ 3434.647930] Key type ._llcrypt registered [ 3434.651724] Key type .llcrypt registered [ 3434.814830] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3435.877105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3435.889143] alg: No test for adler32 (adler32-zlib) [ 3437.028535] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 3437.407542] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3439.055170] Key type lgssc registered [ 3440.149195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3447.749023] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3447.783226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3463.329733] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3468.447272] LustreError: 108952:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 3495.135355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3501.345407] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3511.805968] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3519.485690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3519.895306] LustreError: MGC192.168.202.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3519.908314] LustreError: 109977:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3519.927863] LustreError: 109977:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3519.936789] LustreError: 109977:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3520.067170] Lustre: server umount lustre-MDT0000 complete [ 3520.069510] LustreError: 109977:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3540.448157] Lustre: 108288:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787118496/real 1787118496] req@000000004d9d37c7 x1873929472839040/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787118512 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3540.471136] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3540.502325] LustreError: 108964:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3561.236431] Lustre: server umount lustre-OST0000 complete [ 3572.320200] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3581.613582] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3618.861670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.285507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3619.324847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3619.422650] Lustre: lustre-MDT0000: new disk, initializing [ 3619.549256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.563864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3624.951391] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3638.056498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3638.299314] Lustre: 112774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3638.465034] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3638.472674] Lustre: Skipped 1 previous similar message [ 3638.618487] Lustre: lustre-MDT0001: new disk, initializing [ 3638.765900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3638.826377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3638.838462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3644.141269] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3656.144963] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3663.981909] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3672.772868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3673.106061] Lustre: lustre-OST0000: new disk, initializing [ 3673.115718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3673.130664] Lustre: 114743:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3673.288487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3679.676181] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3682.358990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3682.362487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3682.460513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3689.223370] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3695.227710] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3695.576092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3702.180314] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3702.339842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3707.874905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3707.876564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3707.893981] LustreError: Skipped 1 previous similar message [ 3707.894841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3707.917472] Lustre: Skipped 1 previous similar message [ 3710.654651] Lustre: server umount lustre-OST0000 complete [ 3715.552313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3715.554853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3715.580463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3715.584030] Lustre: Skipped 1 previous similar message [ 3718.114055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.116943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3718.146849] Lustre: Skipped 1 previous similar message [ 3723.234133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3723.259879] Lustre: Skipped 1 previous similar message [ 3727.329432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3727.339964] Lustre: Skipped 1 previous similar message [ 3729.378944] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3729.564103] Lustre: server umount lustre-MDT0000 complete [ 3734.280967] LustreError: 114742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118706 with bad export cookie 986396122136193741 [ 3734.281938] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3734.293375] LustreError: 114742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3734.800736] Lustre: server umount lustre-MDT0001 complete [ 3748.044609] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 01:51:56 (1787118716) [ 3752.457531] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3772.213813] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3784.455296] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3819.594613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.976939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3820.045808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3820.218868] Lustre: lustre-MDT0000: new disk, initializing [ 3820.319402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3820.356535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3826.065925] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3839.857862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.129759] Lustre: 119651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3840.217885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3840.228773] Lustre: Skipped 1 previous similar message [ 3840.357654] Lustre: lustre-MDT0001: new disk, initializing [ 3840.573542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3840.627456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3840.643050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3845.488022] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3855.680859] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3861.125535] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3867.559628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3867.967834] Lustre: lustre-OST0000: new disk, initializing [ 3867.972500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3867.979078] Lustre: 121618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3868.053072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.881853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3873.897582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3874.078537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3874.918430] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3884.532136] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3889.684642] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3889.882946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3894.769291] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3894.956229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3899.873270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3899.876796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3899.885826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3899.887499] Lustre: Skipped 1 previous similar message [ 3899.900212] Lustre: Skipped 1 previous similar message [ 3911.135121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3911.328708] Lustre: server umount lustre-OST0000 complete [ 3917.282182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3917.288801] LustreError: Skipped 1 previous similar message [ 3917.294209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3917.308607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3917.315545] Lustre: Skipped 4 previous similar messages [ 3921.620875] Lustre: server umount lustre-MDT0000 complete [ 3925.252093] LustreError: 119643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118897 with bad export cookie 986396122136194630 [ 3925.258521] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3925.262263] LustreError: 119643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3925.585263] Lustre: server umount lustre-MDT0001 complete [ 3934.379855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.969046] LustreError: 123837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3934.986685] LustreError: 123837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3935.039236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3939.012353] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3940.321522] LustreError: 123837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3940.345252] LustreError: 123837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3944.423687] LustreError: 123838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3946.434162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3950.509948] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3958.665819] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3964.181366] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3969.801683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3976.956840] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3983.703564] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3990.816138] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3995.110025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3995.113242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3995.129382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3995.148505] Lustre: Skipped 2 previous similar messages [ 3995.162069] Lustre: Skipped 2 previous similar messages [ 4000.874110] Lustre: server umount lustre-OST0000 complete [ 4019.167204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4019.343848] Lustre: server umount lustre-MDT0000 complete [ 4020.710492] LustreError: 123837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4020.729029] LustreError: 123837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4022.682920] LustreError: 123819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118994 with bad export cookie 986396122136195512 [ 4022.685520] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4022.691898] LustreError: 123819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4023.069425] Lustre: server umount lustre-MDT0001 complete [ 4032.691996] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 4035.300222] Key type lgssc unregistered [ 4035.574729] LNet: 127467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4035.583043] LNetError: 127467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4035.606535] LNet: Removed LNI 192.168.202.102@tcp [ 4036.417473] Key type .llcrypt unregistered [ 4036.419382] Key type ._llcrypt unregistered [ 4052.419894] Key type ._llcrypt registered [ 4052.421935] Key type .llcrypt registered [ 4052.511058] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 4066.528458] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4067.693807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4067.707381] alg: No test for adler32 (adler32-zlib) [ 4068.761482] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 4069.051483] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 4070.767212] Key type lgssc registered [ 4071.892396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4100.610879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4100.639864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.944318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4101.969198] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4102.051297] Lustre: lustre-MDT0000: new disk, initializing [ 4102.107866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.127526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4105.556460] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4115.046336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.181991] Lustre: 130619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4115.296803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4115.299776] Lustre: Skipped 1 previous similar message [ 4115.379317] Lustre: lustre-MDT0001: new disk, initializing [ 4115.459087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4115.489706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4115.507505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4119.060878] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4128.058427] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4134.184627] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4142.729523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4142.991633] Lustre: lustre-OST0000: new disk, initializing [ 4142.994862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4143.000249] Lustre: 132587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4143.097396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4150.479796] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4150.848512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4150.858453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4150.947834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4158.966890] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4163.072655] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4163.251769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4166.695779] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4166.866579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4171.753306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4171.760393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4171.762094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4171.773110] Lustre: Skipped 1 previous similar message [ 4176.870433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4176.874295] Lustre: Skipped 1 previous similar message [ 4182.011042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4182.021529] Lustre: Skipped 2 previous similar messages [ 4183.011952] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4183.197776] Lustre: server umount lustre-OST0000 complete [ 4187.106195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.117662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.130183] Lustre: Skipped 1 previous similar message [ 4187.135021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4187.160666] Lustre: Skipped 1 previous similar message [ 4192.232330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4192.241994] Lustre: Skipped 1 previous similar message [ 4192.871573] Lustre: server umount lustre-MDT0000 complete [ 4197.005947] LustreError: 130613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119168 with bad export cookie 18207019953852852688 [ 4197.012276] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4197.020137] LustreError: 130613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4197.306642] Lustre: server umount lustre-MDT0001 complete [ 4205.893411] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 01:59:35 (1787119175) [ 4212.981398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.446409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.118536] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4218.852350] LustreError: 135002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4218.859019] LustreError: 135002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4223.971464] LustreError: 135001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4225.825976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.109973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4230.001736] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4236.167911] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4240.908920] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4243.425602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.437537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4249.108107] Lustre: server umount lustre-MDT0000 complete [ 4251.623061] LustreError: 136287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4251.641643] LustreError: 136287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4252.798729] LustreError: 134981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119224 with bad export cookie 18207019953852853570 [ 4252.809877] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4253.185420] Lustre: server umount lustre-MDT0001 complete [ 4261.785050] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:00:31 (1787119231) [ 4268.666193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4284.257460] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4289.375589] LustreError: 137416:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 4315.103381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4321.202325] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4328.950721] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4331.214582] Lustre: server umount lustre-OST0000 complete [ 4338.662409] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:01:48 (1787119308) [ 4344.956723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4345.569572] LustreError: 138872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4345.755696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4350.625517] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4350.949349] LustreError: 138873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.167421] LustreError: 138872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.199266] LustreError: 138872:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4360.438280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.119452] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4375.612174] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4382.909567] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4391.309984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4391.741426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4391.750171] Lustre: Skipped 1 previous similar message [ 4398.236503] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4405.489541] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4425.701295] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4425.702763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4425.710341] Lustre: Skipped 3 previous similar messages [ 4425.720017] Lustre: Skipped 5 previous similar messages [ 4429.537915] Lustre: server umount lustre-MDT0000 complete [ 4430.818880] LustreError: 140669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4430.842105] LustreError: 140669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4433.815504] LustreError: 138853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119405 with bad export cookie 18207019953852854235 [ 4433.827059] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4433.836393] LustreError: 138853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4434.234630] Lustre: server umount lustre-MDT0001 complete [ 4449.085466] Lustre: server umount lustre-OST0000 complete [ 4457.883230] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:03:47 (1787119427) [ 4464.166427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.519720] LustreError: 142548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.583133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4468.328579] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4476.127053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4481.276751] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4489.969844] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4496.094528] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4502.420731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4508.394323] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4510.013445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4517.358249] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4521.817816] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4522.029455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4527.006179] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4527.233782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4530.657167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4530.671669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4530.697878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4535.303354] Lustre: server umount lustre-OST0000 complete [ 4541.920939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4541.940670] Lustre: Skipped 1 previous similar message [ 4545.452090] LustreError: 145861:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000c19ae355) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4545.796793] Lustre: server umount lustre-MDT0000 complete [ 4547.044200] LustreError: 144701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4547.094517] LustreError: 144701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4550.143966] LustreError: 142530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119522 with bad export cookie 18207019953852855313 [ 4550.161176] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4550.165782] LustreError: 142530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4550.431148] Lustre: server umount lustre-MDT0001 complete [ 4561.811246] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:05:31 (1787119531) [ 4569.757899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4585.311949] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4590.431610] LustreError: 146707:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 4616.159746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4616.166594] Lustre: Skipped 2 previous similar messages [ 4623.000853] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4631.501914] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4638.597616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.040932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4643.552501] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4651.710869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.365756] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4666.200639] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4672.371457] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4678.600638] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4678.980491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4684.619567] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4684.843955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4688.362090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4688.381442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.408251] Lustre: Skipped 1 previous similar message [ 4688.421976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4688.434277] Lustre: Skipped 4 previous similar messages [ 4693.750146] Lustre: server umount lustre-MDT0000 complete [ 4696.556796] LustreError: 148745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4696.574833] LustreError: 148745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4698.386829] LustreError: 146715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119670 with bad export cookie 18207019953852856335 [ 4698.415335] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4698.419552] LustreError: 146715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4698.952203] Lustre: server umount lustre-MDT0001 complete [ 4713.476711] Lustre: server umount lustre-OST0000 complete [ 4723.355229] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:08:12 (1787119692) [ 4731.572101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4747.361658] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4752.483753] LustreError: 150900:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 4784.017918] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4792.816692] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4800.026183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4800.946049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4805.640149] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4814.636660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4819.737331] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4828.116106] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4835.258446] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4841.553264] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4841.755180] Lustre: lustre-OST0001: new disk, initializing [ 4841.760783] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4841.769961] Lustre: 153777:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4848.440608] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4850.713059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4850.733011] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4850.852879] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4856.851159] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4861.733393] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4861.967862] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4866.449972] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4866.744163] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4871.651946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4871.659985] Lustre: Skipped 2 previous similar messages [ 4871.666527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4871.675821] Lustre: Skipped 4 previous similar messages [ 4874.953677] Lustre: server umount lustre-OST0000 complete [ 4881.402640] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4885.138061] Lustre: server umount lustre-OST0001 complete [ 4892.153252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4892.156625] LustreError: Skipped 1 previous similar message [ 4903.392516] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4903.641844] Lustre: server umount lustre-MDT0000 complete [ 4907.300432] LustreError: 150908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119879 with bad export cookie 18207019953852857287 [ 4907.304494] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4907.309806] LustreError: 150908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4907.656642] Lustre: server umount lustre-MDT0001 complete [ 4932.291798] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:11:41 (1787119901) [ 4933.861847] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4935.680475] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4937.378672] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:11:47 (1787119907) [ 4944.282841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4944.519710] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4944.547980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4944.758889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4944.765769] Lustre: Skipped 6 previous similar messages [ 4949.186866] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4957.348123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4957.525967] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4957.584694] Lustre: 157335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4962.515912] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4969.405632] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4974.752666] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4980.946391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4981.276674] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4987.231938] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4990.112791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4995.014905] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4999.859452] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5000.036712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5004.343847] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5004.580636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5010.914558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5010.943431] Lustre: Skipped 6 previous similar messages [ 5012.645344] Lustre: server umount lustre-OST0000 complete [ 5037.440375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5044.401684] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5051.368160] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.395295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.398621] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5051.441293] Lustre: Skipped 1 previous similar message [ 5052.600861] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5057.051301] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5057.283973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5062.056738] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5062.226740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5067.262644] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9565d0428800.ost_server_uuid 50 [ 5068.675877] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9565d0428800.ost_server_uuid in FULL state after 0 sec [ 5068.925317] LustreError: 159872:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.2@tcp) returned error from blocking AST (req@00000000cbe04b64 x1873930135802496 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d25061a7/0xfcac542edfd94521 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.2@tcp remote: 0x8d8fc99c4dc2e4a6 expref: 5 pid: 157343 timeout: 5168 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5068.961520] LustreError: lustre-MDT0000: A client on nid 192.168.202.2@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5068.972313] LustreError: 157333:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.2@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d25061a7/0xfcac542edfd94521 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.2@tcp remote: 0x8d8fc99c4dc2e4a6 expref: 6 pid: 157343 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5074.916278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5079.607814] Lustre: server umount lustre-OST0000 complete [ 5089.683726] Lustre: server umount lustre-MDT0000 complete [ 5092.326808] LustreError: 161820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5092.351541] LustreError: 161820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 5094.141478] LustreError: 157328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120066 with bad export cookie 18207019953852858463 [ 5094.153201] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5094.164703] LustreError: 157328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5100.713556] Lustre: server umount lustre-MDT0001 complete [ 5113.084817] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5115.233904] Key type lgssc unregistered [ 5115.469073] LNet: 163459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5115.497155] LNetError: 163459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5115.516896] LNet: Removed LNI 192.168.202.102@tcp [ 5116.351196] Key type .llcrypt unregistered [ 5116.357846] Key type ._llcrypt unregistered [ 5146.773635] Key type ._llcrypt registered [ 5146.779225] Key type .llcrypt registered [ 5146.906494] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5147.919899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5147.972599] alg: No test for adler32 (adler32-zlib) [ 5149.166181] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 5149.414324] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5151.079591] Key type lgssc registered [ 5152.114978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5160.996719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5161.032843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5162.519493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.972792] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5172.712696] LustreError: 164874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.734464] LustreError: 164874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5176.574254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.111325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5182.231880] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5189.332444] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5195.886522] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5203.383236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5203.908855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.069730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5211.523658] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5221.521820] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5226.920723] Lustre: Failing over lustre-MDT0000 [ 5227.187534] Lustre: server umount lustre-MDT0000 complete [ 5227.493761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5227.495859] LustreError: 164869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5227.505369] Lustre: Skipped 2 previous similar messages [ 5227.522429] LustreError: 164869:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5228.536417] LustreError: 164871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5232.609318] LustreError: 164875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5232.614713] LustreError: 164875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5237.729138] LustreError: 164870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.745418] LustreError: 164870:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5239.542525] LustreError: 164856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.2@tcp arrived at 1787120211 with bad export cookie 10226701836737563506 [ 5239.563116] LustreError: 164856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5243.872330] Lustre: 164187:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120199/real 1787120199] req@0000000014263afb x1873931268022912/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787120215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5243.909521] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5246.279338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5248.992935] LustreError: 165603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5249.006041] LustreError: 165603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5253.166573] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8dec8b231a93cbdb [ 5253.171979] Lustre: MGC192.168.202.102@tcp: Connection restored to 0@lo (at 0@lo) [ 5253.539190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5253.610580] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5258.107387] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5258.734206] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5258.743573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5260.431895] Lustre: lustre-MDT0000: Denying connection for new client bbc09040-6537-427a-8c84-659c1802d697 (at 192.168.202.2@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5268.974171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5268.999451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5273.575881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5273.584429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.601613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5279.202132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5279.208248] Lustre: Skipped 3 previous similar messages [ 5283.295631] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5283.552373] Lustre: server umount lustre-OST0000 complete [ 5287.448052] LustreError: 168609:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5287.462791] Lustre: 167958:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5287.476414] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5287.500114] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5287.510413] LustreError: 167958:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ea308fde x1873931268044416/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5287.521634] LustreError: 167958:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5287.527214] LustreError: 167958:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5287.533454] Lustre: lustre-MDT0000: Recovery over after 0:29, of 2 clients 0 recovered and 2 were evicted. [ 5287.964991] Lustre: server umount lustre-MDT0000 complete [ 5289.441838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5289.473767] LustreError: 164870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5289.515884] LustreError: 164870:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5291.641879] LustreError: 164854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120263 with bad export cookie 10226701836737563611 [ 5291.643873] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5291.660373] LustreError: 164854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5291.660386] LustreError: 164183:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5294.098411] Lustre: server umount lustre-MDT0001 complete [ 5305.124339] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5307.446498] Key type lgssc unregistered [ 5307.732451] LNet: 169362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5307.747422] LNetError: 169362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5307.781597] LNet: Removed LNI 192.168.202.102@tcp [ 5308.573412] Key type .llcrypt unregistered [ 5308.575666] Key type ._llcrypt unregistered [ 5330.487212] Key type ._llcrypt registered [ 5330.493479] Key type .llcrypt registered [ 5330.620261] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5331.588920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5331.611663] alg: No test for adler32 (adler32-zlib) [ 5332.776611] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 5333.080261] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5334.807269] Key type lgssc registered [ 5335.769136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5342.921556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5342.936432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5344.284380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5349.559513] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5354.477707] LustreError: 170771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5354.500218] LustreError: 170771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5358.450348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.782450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5362.689272] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5370.467482] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5376.070278] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5382.608990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5383.005385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5388.555957] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5389.224476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5394.569672] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5404.643834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5404.644382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5404.655267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5404.656061] LustreError: Skipped 1 previous similar message [ 5404.664178] Lustre: Skipped 1 previous similar message [ 5407.187642] Lustre: server umount lustre-OST0000 complete [ 5414.911047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5414.921180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5414.925433] Lustre: Skipped 1 previous similar message [ 5414.928629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5416.884787] Lustre: server umount lustre-MDT0000 complete [ 5420.002829] LustreError: 170772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5420.017022] LustreError: 170772:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5420.143469] LustreError: 170751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120392 with bad export cookie 14183100031609984938 [ 5420.157993] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5420.580522] Lustre: server umount lustre-MDT0001 complete [ 5430.459879] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5432.926476] Key type lgssc unregistered [ 5433.239865] LNet: 174305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5433.245789] LNetError: 174305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5433.266324] LNet: Removed LNI 192.168.202.102@tcp [ 5434.047388] Key type .llcrypt unregistered [ 5434.049967] Key type ._llcrypt unregistered [ 5460.528256] Key type ._llcrypt registered [ 5460.529523] Key type .llcrypt registered [ 5460.609754] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5461.678712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5461.710303] alg: No test for adler32 (adler32-zlib) [ 5462.703252] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 5462.887982] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5464.511289] Key type lgssc registered [ 5465.554321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5472.241919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5472.258780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5473.629657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5477.482552] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5484.005432] LustreError: 175987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5484.023706] LustreError: 175987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5484.486214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5484.842709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5488.698522] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5495.573318] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5500.619688] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5506.406174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5506.770922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5512.795227] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5519.100966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5519.598368] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5525.473583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5525.488257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.506727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5529.572400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5529.578929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5529.579658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.587519] Lustre: Skipped 1 previous similar message [ 5534.690864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5539.295253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5539.497679] Lustre: server umount lustre-OST0000 complete [ 5544.933509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5544.945351] Lustre: Skipped 1 previous similar message [ 5544.946157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5544.967095] Lustre: Skipped 2 previous similar messages [ 5549.264831] Lustre: server umount lustre-MDT0000 complete [ 5550.055691] LustreError: 177226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5550.077101] LustreError: 177226:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5552.771947] LustreError: 175968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120524 with bad export cookie 11998242052009912753 [ 5552.789108] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5553.148653] Lustre: server umount lustre-MDT0001 complete [ 5564.382840] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5566.664663] Key type lgssc unregistered [ 5567.009817] LNet: 179519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5567.023617] LNetError: 179519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5567.054654] LNet: Removed LNI 192.168.202.102@tcp [ 5567.955226] Key type .llcrypt unregistered [ 5567.957837] Key type ._llcrypt unregistered [ 5588.561329] Key type ._llcrypt registered [ 5588.563563] Key type .llcrypt registered [ 5588.667716] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5589.986668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5590.020429] alg: No test for adler32 (adler32-zlib) [ 5591.231724] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 5591.540960] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5593.207979] Key type lgssc registered [ 5594.321100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5605.965741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5605.995531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5607.434180] Lustre: *** cfs_fail_loc=135, val=0*** [ 5607.564500] LustreError: 181022:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5607.574988] LustreError: 181022:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.102@tcp: cfg command failed: rc = -2 [ 5607.586837] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5607.586837] [ 5607.604943] LustreError: MGC192.168.202.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5607.621348] LustreError: 180984:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5607.625957] LustreError: 180984:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5607.635541] LustreError: 180984:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5607.650816] LustreError: 180984:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5607.772536] Lustre: server umount lustre-MDT0000 complete [ 5607.779108] LustreError: 180984:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5614.310946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.638237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5618.585873] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5624.802345] LustreError: 181392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5624.820556] LustreError: 181392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5626.471058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5626.878588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5630.990953] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5637.199265] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5642.059277] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5644.256660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5644.259554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5644.265828] Lustre: Skipped 1 previous similar message [ 5644.272546] Lustre: Skipped 1 previous similar message [ 5649.377826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5649.382538] Lustre: Skipped 1 previous similar message [ 5651.426754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5651.436469] Lustre: Skipped 1 previous similar message [ 5656.546770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5656.569074] Lustre: Skipped 1 previous similar message [ 5658.079185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5658.307528] Lustre: server umount lustre-MDT0000 complete [ 5661.673907] LustreError: 181392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.705742] LustreError: 181392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5661.953229] LustreError: 182127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120633 with bad export cookie 140836597945523428 [ 5661.967212] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5662.241115] Lustre: server umount lustre-MDT0001 complete [ 5672.804507] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5675.327041] Key type lgssc unregistered [ 5675.591746] LNet: 183713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5675.598247] LNetError: 183713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5675.610918] LNet: Removed LNI 192.168.202.102@tcp [ 5676.358662] Key type .llcrypt unregistered [ 5676.362303] Key type ._llcrypt unregistered [ 5696.586777] Key type ._llcrypt registered [ 5696.588569] Key type .llcrypt registered [ 5696.706655] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5697.072515] Key type .llcrypt unregistered [ 5697.082511] Key type ._llcrypt unregistered [ 5714.601733] Key type ._llcrypt registered [ 5714.611765] Key type .llcrypt registered [ 5714.755074] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5716.261981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5716.312214] alg: No test for adler32 (adler32-zlib) [ 5717.439898] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 5717.748803] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5719.423602] Key type lgssc registered [ 5720.495611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5727.709896] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5727.732888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5743.327902] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5748.453058] LustreError: 185849:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 5775.199231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5780.557938] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5787.889498] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5793.128159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5793.538885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5793.609601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5797.694438] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5798.880047] LustreError: 185874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.901019] LustreError: 185874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5804.013666] LustreError: 185875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.422649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5805.817515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5810.200712] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5817.250227] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5823.343766] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5825.983643] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5839.849131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5839.859228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.873343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5840.876087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5840.893631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5844.035384] Lustre: server umount lustre-OST0000 complete [ 5851.107180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.128636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5851.138736] Lustre: Skipped 2 previous similar messages [ 5854.847200] Lustre: server umount lustre-MDT0000 complete [ 5856.225740] LustreError: 186890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.259645] LustreError: 186890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5858.381823] LustreError: 185855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120830 with bad export cookie 14551983354571421119 [ 5858.383542] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5858.394697] LustreError: 185855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5858.687678] Lustre: server umount lustre-MDT0001 complete [ 5869.075365] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5871.660848] Key type lgssc unregistered [ 5871.952794] LNet: 189863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5871.963948] LNetError: 189863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5871.995163] LNet: Removed LNI 192.168.202.102@tcp [ 5872.800198] Key type .llcrypt unregistered [ 5872.802083] Key type ._llcrypt unregistered [ 5891.946227] Key type ._llcrypt registered [ 5891.948907] Key type .llcrypt registered [ 5892.078723] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5893.179049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5893.201378] alg: No test for adler32 (adler32-zlib) [ 5894.373633] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 5894.758442] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5896.439386] Key type lgssc registered [ 5897.481605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5904.432246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5904.446378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5905.812979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5909.210937] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5915.116321] LustreError: 191255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5915.132489] LustreError: 191255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5917.654948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5917.963959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5921.893760] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5928.500831] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5933.989032] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5940.137963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5940.757690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5946.312345] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5951.030948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5954.211754] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5959.064471] Lustre: Failing over lustre-MDT0000 [ 5959.149242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5959.152353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5959.163588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5959.263464] Lustre: server umount lustre-MDT0000 complete [ 5961.185857] LustreError: 191255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.206638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.310186] LustreError: 191252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.335365] LustreError: 191252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5971.427985] LustreError: 191251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5971.451255] LustreError: 191251:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5976.547498] LustreError: 191255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5976.580758] LustreError: 191255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5977.568201] Lustre: 190567:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120933/real 1787120933] req@00000000321e2139 x1873932049211904/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787120949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5977.591258] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5978.646802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5987.813325] LustreError: 191256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5987.852235] LustreError: 191256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5988.224304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5988.297954] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5992.953790] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5993.197776] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5993.445882] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5993.481260] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5993.569606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5997.494646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6005.400348] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6013.932510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6013.935306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6013.949850] Lustre: Skipped 2 previous similar messages [ 6013.959966] Lustre: Skipped 1 previous similar message [ 6015.446723] Lustre: server umount lustre-OST0000 complete [ 6024.165415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6024.174471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6024.177916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6024.183014] Lustre: Skipped 1 previous similar message [ 6025.472461] Lustre: server umount lustre-MDT0000 complete [ 6029.286034] LustreError: 191256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6029.301904] LustreError: 191256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6029.776391] LustreError: 193100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121001 with bad export cookie 5159954343748490650 [ 6029.793623] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6029.798644] LustreError: 193100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6030.272966] Lustre: server umount lustre-MDT0001 complete [ 6040.699565] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6043.232368] Key type lgssc unregistered [ 6043.550974] LNet: 196148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6043.562749] LNetError: 196148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6043.591772] LNet: Removed LNI 192.168.202.102@tcp [ 6044.408195] Key type .llcrypt unregistered [ 6044.410259] Key type ._llcrypt unregistered [ 6070.548260] Key type ._llcrypt registered [ 6070.557803] Key type .llcrypt registered [ 6070.752486] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6072.637903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6072.781209] alg: No test for adler32 (adler32-zlib) [ 6073.822809] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 6073.955542] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6075.577320] Key type lgssc registered [ 6077.380797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6087.571429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6087.610964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.185080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6094.247896] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6099.433495] LustreError: 197570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.463063] LustreError: 197570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6100.450732] LustreError: 197569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.651484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6104.067419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6108.042487] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6115.411977] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6120.916276] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6127.864900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6128.406486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6134.889889] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6135.685074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6145.776544] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6149.832790] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6153.323690] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6174.769495] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6185.968751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6185.988262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6186.021967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6186.983453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6186.984520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6186.999398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6187.041647] Lustre: Skipped 1 previous similar message [ 6189.203841] Lustre: server umount lustre-OST0000 complete [ 6196.203609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6196.214742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.240844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6199.537587] Lustre: server umount lustre-MDT0000 complete [ 6202.336761] LustreError: 200117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6202.365354] LustreError: 200117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6203.878138] LustreError: 197552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121175 with bad export cookie 859485990899365443 [ 6203.879327] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.915070] LustreError: 197552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6204.546543] Lustre: server umount lustre-MDT0001 complete [ 6216.944027] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6220.677425] Key type lgssc unregistered [ 6221.085124] LNet: 201437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6221.095992] LNetError: 201437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6221.127471] LNet: Removed LNI 192.168.202.102@tcp [ 6222.187768] Key type .llcrypt unregistered [ 6222.190065] Key type ._llcrypt unregistered [ 6244.065906] Key type ._llcrypt registered [ 6244.068620] Key type .llcrypt registered [ 6244.229334] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6245.970318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6246.020083] alg: No test for adler32 (adler32-zlib) [ 6247.168088] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 6247.520084] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6249.255181] Key type lgssc registered [ 6250.965758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6260.489466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6260.516728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6261.906656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6267.295329] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6271.458958] LustreError: 202705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6271.472923] LustreError: 202705:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6276.430132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6276.579107] LustreError: 202704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6277.004800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6281.266872] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6288.967964] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6294.628345] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6301.759825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6302.215540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6306.363617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6308.661646] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6316.302837] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6322.900565] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6336.374792] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6345.158522] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6351.118601] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6362.599397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6362.602837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6362.608652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6362.631894] Lustre: Skipped 1 previous similar message [ 6362.661745] Lustre: Skipped 1 previous similar message [ 6367.714330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6372.835943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6372.847989] Lustre: Skipped 2 previous similar messages [ 6374.879306] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6375.218469] Lustre: server umount lustre-OST0000 complete [ 6379.488616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6379.496892] LustreError: Skipped 1 previous similar message [ 6379.509577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6379.543655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6383.079055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6388.195513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6388.210429] Lustre: Skipped 3 previous similar messages [ 6393.311216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6393.610316] Lustre: server umount lustre-MDT0000 complete [ 6397.411177] LustreError: 202684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121369 with bad export cookie 1180166521480783735 [ 6397.414058] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6397.419542] LustreError: 202684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6397.871310] Lustre: server umount lustre-MDT0001 complete [ 6410.713323] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6413.604291] Key type lgssc unregistered [ 6413.967560] LNet: 207630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6413.973725] LNetError: 207630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6413.990627] LNet: Removed LNI 192.168.202.102@tcp [ 6414.972986] Key type .llcrypt unregistered [ 6414.975208] Key type ._llcrypt unregistered [ 6439.047117] Key type ._llcrypt registered [ 6439.050252] Key type .llcrypt registered [ 6439.227359] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6440.710714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6440.740819] alg: No test for adler32 (adler32-zlib) [ 6441.828255] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 6442.107421] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6443.815741] Key type lgssc registered [ 6444.887695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6451.745114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6451.813407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6453.249939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6457.885459] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6463.458211] LustreError: 209044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.477778] LustreError: 209044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6467.829173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6468.468059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6474.632197] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6483.593862] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6489.893098] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6498.086735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6498.342098] LustreError: 210866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6498.664529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6503.932130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6506.002374] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6513.911497] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6533.602154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6533.619357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6533.640124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6537.335935] Lustre: server umount lustre-OST0000 complete [ 6543.841539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.868115] Lustre: Skipped 1 previous similar message [ 6543.883137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6543.886017] Lustre: Skipped 2 previous similar messages [ 6548.579327] Lustre: server umount lustre-MDT0000 complete [ 6548.965479] LustreError: 209045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6549.008906] LustreError: 209045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6553.069837] LustreError: 209044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6553.114540] LustreError: 209044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6553.858243] LustreError: 209025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121525 with bad export cookie 15198921525735312598 [ 6553.874248] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6554.424821] Lustre: server umount lustre-MDT0001 complete [ 6568.373386] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6571.752125] Key type lgssc unregistered [ 6572.070365] LNet: 212729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6572.078440] LNetError: 212729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6572.093454] LNet: Removed LNI 192.168.202.102@tcp [ 6573.075125] Key type .llcrypt unregistered [ 6573.077528] Key type ._llcrypt unregistered [ 6594.961863] Key type ._llcrypt registered [ 6594.964232] Key type .llcrypt registered [ 6595.063671] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6596.178491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6596.264388] alg: No test for adler32 (adler32-zlib) [ 6597.441598] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 6597.696230] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6599.391192] Key type lgssc registered [ 6600.461668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6609.013810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6609.038595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6610.537860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6615.122173] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6620.642680] LustreError: 213998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6620.657220] LustreError: 213998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6623.014686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6623.384984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6628.218189] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6635.883728] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6641.617719] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6648.222904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6648.727138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6655.916777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6657.057800] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6664.703545] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6678.496741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6678.509242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6678.530079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6678.554580] Lustre: Skipped 1 previous similar message [ 6683.631969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6683.636820] Lustre: Skipped 1 previous similar message [ 6683.934418] Lustre: server umount lustre-OST0000 complete [ 6688.738041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6688.747642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6688.753907] Lustre: Skipped 1 previous similar message [ 6688.762602] Lustre: Skipped 1 previous similar message [ 6693.859522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6693.871826] Lustre: Skipped 1 previous similar message [ 6694.147906] Lustre: server umount lustre-MDT0000 complete [ 6697.623834] LustreError: 213977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121669 with bad export cookie 17331293987891401088 [ 6697.636909] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6697.637354] LustreError: 213977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6697.955090] Lustre: server umount lustre-MDT0001 complete [ 6709.785484] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6713.246633] Key type lgssc unregistered [ 6713.570850] LNet: 217677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6713.581441] LNetError: 217677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6713.593706] LNet: Removed LNI 192.168.202.102@tcp [ 6714.689850] Key type .llcrypt unregistered [ 6714.694895] Key type ._llcrypt unregistered [ 6735.371342] Key type ._llcrypt registered [ 6735.374759] Key type .llcrypt registered [ 6735.482695] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6736.642854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6736.685231] alg: No test for adler32 (adler32-zlib) [ 6737.841128] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 6738.237797] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6740.039195] Key type lgssc registered [ 6741.330225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6751.009782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6751.041331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.388262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6757.631371] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6762.977847] LustreError: 218945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6762.995720] LustreError: 218945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6767.280754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.611067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6771.869682] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6779.474384] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6786.001970] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6794.210602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6795.032373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6797.112604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6803.312723] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6812.888664] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6823.023661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6823.093822] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6823.308616] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6830.894398] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6840.708399] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6842.735205] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6854.672706] Lustre: Permanently deactivating lustre-OST0001 [ 6854.717178] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6860.450806] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6867.883182] Lustre: Permanently reactivating lustre-OST0001 [ 6867.894801] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6867.899366] Lustre: Skipped 2 previous similar messages [ 6869.236213] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6869.256509] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6869.265329] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6869.282329] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6869.322704] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6879.713678] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6879.714938] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6879.725844] LustreError: Skipped 1 previous similar message [ 6879.726623] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6879.763494] Lustre: Skipped 2 previous similar messages [ 6881.267746] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6881.276853] Lustre: Skipped 2 previous similar messages [ 6883.337028] Lustre: server umount lustre-OST0001 complete [ 6886.374088] LustreError: 220770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6886.397777] LustreError: 220770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6889.442066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6889.444108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6889.455647] LustreError: Skipped 1 previous similar message [ 6889.466783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6889.471877] Lustre: Skipped 1 previous similar message [ 6889.487109] Lustre: Skipped 1 previous similar message [ 6891.498090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6891.512381] LustreError: 223004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6891.524326] LustreError: 223004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6893.580182] Lustre: server umount lustre-OST0000 complete [ 6900.723965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6900.730402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6900.746611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6900.751251] Lustre: Skipped 1 previous similar message [ 6903.957749] Lustre: server umount lustre-MDT0000 complete [ 6906.859350] LustreError: 218940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6906.889222] LustreError: 218940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6908.137058] LustreError: 218925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121880 with bad export cookie 18344453281858243808 [ 6908.139080] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6908.146671] LustreError: 218925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6908.773478] Lustre: server umount lustre-MDT0001 complete [ 6920.982280] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6924.484983] Key type lgssc unregistered [ 6924.892939] LNet: 223994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6924.898580] LNetError: 223994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6924.912961] LNet: Removed LNI 192.168.202.102@tcp [ 6926.050905] Key type .llcrypt unregistered [ 6926.053701] Key type ._llcrypt unregistered [ 6968.218775] Key type ._llcrypt registered [ 6968.234128] Key type .llcrypt registered [ 6968.413844] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6969.416115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6969.461461] alg: No test for adler32 (adler32-zlib) [ 6971.038522] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 6971.281556] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6972.975522] Key type lgssc registered [ 6974.324107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6983.167109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6983.184922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.494186] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6984.521497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6984.722216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6989.591610] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6999.683247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6999.783273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6999.827306] Lustre: 226199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7000.133683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7004.710494] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7014.209989] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7020.970246] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7028.206093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7028.362427] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7028.549784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7034.054402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7034.855944] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7043.523750] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7046.307983] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7052.035773] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7056.780512] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7063.566650] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7072.626869] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7096.826573] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7096.834298] Lustre: Skipped 2 previous similar messages [ 7134.353189] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7134.359234] Lustre: Skipped 4 previous similar messages [ 7208.104457] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7208.107526] Lustre: Skipped 8 previous similar messages [ 7218.656978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7218.661433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7218.668679] LustreError: Skipped 1 previous similar message [ 7218.672271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7218.705636] Lustre: Skipped 1 previous similar message [ 7223.778867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7223.797842] Lustre: Skipped 1 previous similar message [ 7228.383155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7228.586258] Lustre: server umount lustre-OST0000 complete [ 7234.018472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.021721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7234.033952] Lustre: Skipped 1 previous similar message [ 7234.053176] Lustre: Skipped 1 previous similar message [ 7239.093058] Lustre: server umount lustre-MDT0000 complete [ 7243.157141] LustreError: 226189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122215 with bad export cookie 1472947486040077700 [ 7243.162339] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7243.170078] LustreError: 226189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7243.677861] Lustre: server umount lustre-MDT0001 complete [ 7258.954217] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7262.121185] Key type lgssc unregistered [ 7262.601451] LNet: 230993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7262.624257] LNetError: 230993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7262.646533] LNet: Removed LNI 192.168.202.102@tcp [ 7263.931249] Key type .llcrypt unregistered [ 7263.936875] Key type ._llcrypt unregistered [ 7288.885762] Key type ._llcrypt registered [ 7288.891475] Key type .llcrypt registered [ 7289.042417] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7290.434547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7290.486950] alg: No test for adler32 (adler32-zlib) [ 7291.668732] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 7291.984632] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7293.743974] Key type lgssc registered [ 7295.028417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7302.220784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7302.236528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.735663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7308.647399] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7313.897214] LustreError: 232407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7313.913682] LustreError: 232407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7317.588488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7318.230443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7322.490704] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7330.087604] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7336.363353] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7342.427946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7342.746094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7348.370441] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7348.925893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7355.236793] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7374.816438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7374.819868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7374.824606] LustreError: Skipped 1 previous similar message [ 7374.830176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7374.848324] Lustre: Skipped 1 previous similar message [ 7375.914485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7375.928427] Lustre: Skipped 1 previous similar message [ 7380.243250] Lustre: server umount lustre-OST0000 complete [ 7387.112332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7387.124580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7387.153759] Lustre: Skipped 1 previous similar message [ 7387.173571] Lustre: Skipped 1 previous similar message [ 7392.227750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7392.236323] Lustre: Skipped 2 previous similar messages [ 7397.345253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7397.352996] Lustre: Skipped 1 previous similar message [ 7399.391168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7399.640957] Lustre: server umount lustre-MDT0000 complete [ 7401.442730] LustreError: 232408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7401.469664] LustreError: 232408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7402.477605] LustreError: 232407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7402.505257] LustreError: 232407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7403.100441] LustreError: 232386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122375 with bad export cookie 16025244485231166356 [ 7403.101394] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7403.110226] LustreError: 232386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7403.482253] Lustre: server umount lustre-MDT0001 complete [ 7415.651660] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7418.843083] Key type lgssc unregistered [ 7419.118373] LNet: 236100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7419.125906] LNetError: 236100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7419.150995] LNet: Removed LNI 192.168.202.102@tcp [ 7419.982436] Key type .llcrypt unregistered [ 7419.987615] Key type ._llcrypt unregistered [ 7441.706958] Key type ._llcrypt registered [ 7441.708699] Key type .llcrypt registered [ 7441.807922] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7442.773413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7442.903559] alg: No test for adler32 (adler32-zlib) [ 7443.860129] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 7444.032740] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7445.679204] Key type lgssc registered [ 7446.849399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7718.688469] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7721.972617] Key type lgssc unregistered [ 7722.296029] LNet: 238105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7722.313881] LNetError: 238105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7722.360223] LNet: Removed LNI 192.168.202.102@tcp [ 7723.179966] Key type .llcrypt unregistered [ 7723.181713] Key type ._llcrypt unregistered [ 7752.702836] Key type ._llcrypt registered [ 7752.705762] Key type .llcrypt registered [ 7752.891175] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 7770.857566] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7772.077056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7772.301076] alg: No test for adler32 (adler32-zlib) [ 7773.490097] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 7773.693276] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7775.351190] Key type lgssc registered [ 7776.367887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7805.372853] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7515 sec ========== 02:59:34 (1787122774) [ 7807.410951] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:59:36 (1787122776) === [ 7810.668513] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:59:40 (1787122780) === [ 7858.041144] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7861.880912] Key type lgssc unregistered [ 7862.445664] LNet: 243344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7862.454954] LNetError: 243344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7862.475745] LNet: Removed LNI 192.168.202.102@tcp [ 7863.444605] Key type .llcrypt unregistered [ 7863.446990] Key type ._llcrypt unregistered