[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480840852 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002249] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.010000] LSM: Security Framework initializing [ 0.010041] Yama: becoming mindful. [ 0.011042] SELinux: Initializing. [ 0.012068] *** VALIDATE selinux *** [ 0.020198] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025155] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] *** VALIDATE tmpfs *** [ 0.031155] *** VALIDATE proc *** [ 0.032000] *** VALIDATE cgroup *** [ 0.032013] *** VALIDATE cgroup2 *** [ 0.033248] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035005] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042294] debug: unmapping init [mem 0xffffffff84859000-0xffffffff84860fff] [ 0.044125] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045684] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047022] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051011] ... fixed-purpose events: 3 [ 0.051913] ... event mask: 000000070000000f [ 0.052243] rcu: Hierarchical SRCU implementation. [ 0.054492] smp: Bringing up secondary CPUs ... [ 0.055600] x86: Booting SMP configuration: [ 0.056029] .... node #0, CPUs: #1 #2 #3 [ 0.059424] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.230337] node 0 deferred pages initialised in 166ms [ 0.233212] devtmpfs: initialized [ 0.234432] x86/mm: Memory block size: 128MB [ 0.238150] gcov: version magic: 0x41383552 [ 0.241331] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.245091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.248272] pinctrl core: initialized pinctrl subsystem [ 0.250186] [ 0.250770] ************************************************************* [ 0.253016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.256014] ** ** [ 0.258011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.260012] ** ** [ 0.261010] ** This means that this kernel is built to expose internal ** [ 0.263013] ** IOMMU data structures, which may compromise security on ** [ 0.265010] ** your system. ** [ 0.267012] ** ** [ 0.269014] ** If you see this message and you are not debugging the ** [ 0.272021] ** kernel, report this immediately to your vendor! ** [ 0.274011] ** ** [ 0.276013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.278012] ************************************************************* [ 0.281643] NET: Registered protocol family 16 [ 0.283544] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.286062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.289068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.293181] cpuidle: using governor menu [ 0.294550] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.297400] PCI: Using configuration type 1 for base access [ 0.299129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.309212] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.311020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.313258] cryptd: max_cpu_qlen set to 1000 [ 0.317159] ACPI: Added _OSI(Module Device) [ 0.318011] ACPI: Added _OSI(Processor Device) [ 0.320009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.321007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.324314] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.329207] ACPI: Interpreter enabled [ 0.330053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.331010] ACPI: Using IOAPIC for interrupt routing [ 0.332074] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.334328] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.345403] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.347056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.350034] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.353110] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.358529] acpiphp: Slot [2] registered [ 0.360263] acpiphp: Slot [5] registered [ 0.362211] acpiphp: Slot [6] registered [ 0.363147] acpiphp: Slot [7] registered [ 0.365187] acpiphp: Slot [8] registered [ 0.366174] acpiphp: Slot [9] registered [ 0.368195] acpiphp: Slot [10] registered [ 0.369117] acpiphp: Slot [3] registered [ 0.371138] acpiphp: Slot [4] registered [ 0.373104] acpiphp: Slot [11] registered [ 0.374123] acpiphp: Slot [12] registered [ 0.376111] acpiphp: Slot [13] registered [ 0.377110] acpiphp: Slot [14] registered [ 0.379120] acpiphp: Slot [15] registered [ 0.380136] acpiphp: Slot [16] registered [ 0.382185] acpiphp: Slot [17] registered [ 0.384112] acpiphp: Slot [18] registered [ 0.385100] acpiphp: Slot [19] registered [ 0.387103] acpiphp: Slot [20] registered [ 0.388101] acpiphp: Slot [21] registered [ 0.390104] acpiphp: Slot [22] registered [ 0.391137] acpiphp: Slot [23] registered [ 0.393103] acpiphp: Slot [24] registered [ 0.394105] acpiphp: Slot [25] registered [ 0.396104] acpiphp: Slot [26] registered [ 0.397138] acpiphp: Slot [27] registered [ 0.399111] acpiphp: Slot [28] registered [ 0.401148] acpiphp: Slot [29] registered [ 0.402156] acpiphp: Slot [30] registered [ 0.404198] acpiphp: Slot [31] registered [ 0.406097] PCI host bridge to bus 0000:00 [ 0.408032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.410037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.413042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.416037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.418035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.422042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.424216] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.427153] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.430534] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.441814] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.447050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.449030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.451027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.454027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.456558] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.459816] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.462042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.465825] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.471017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.483017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.488027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.493737] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.499016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.504026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.528017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.538325] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.546026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.553026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.569030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.586262] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.591015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.598030] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.610015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.617488] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.622013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.627012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.636018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.644417] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.652028] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.661028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.679026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.689798] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.702015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.708016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.743016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.753646] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.756410] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.759466] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.761411] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.764286] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.768148] iommu: Default domain type: Passthrough [ 0.772049] SCSI subsystem initialized [ 0.774203] ACPI: bus type USB registered [ 0.775143] usbcore: registered new interface driver usbfs [ 0.777122] usbcore: registered new interface driver hub [ 0.780183] usbcore: registered new device driver usb [ 0.782240] pps_core: LinuxPPS API ver. 1 registered [ 0.784019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.787093] PTP clock support registered [ 0.789231] EDAC MC: Ver: 3.0.0 [ 0.792008] PCI: Using ACPI for IRQ routing [ 0.793941] NetLabel: Initializing [ 0.795011] NetLabel: domain hash size = 128 [ 0.797010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.799087] NetLabel: unlabeled traffic allowed by default [ 0.802140] vgaarb: loaded [ 0.803268] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.805014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.811358] clocksource: Switched to clocksource kvm-clock [ 0.924206] VFS: Disk quotas dquot_6.6.0 [ 0.925399] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.927446] *** VALIDATE ramfs *** [ 0.928328] *** VALIDATE hugetlbfs *** [ 0.929488] pnp: PnP ACPI init [ 0.931401] pnp: PnP ACPI: found 6 devices [ 0.950193] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.952825] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.955039] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.956912] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.959272] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.961651] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.964087] NET: Registered protocol family 2 [ 0.966378] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.970991] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.974373] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.978750] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.982397] TCP: Hash tables configured (established 65536 bind 65536) [ 0.985366] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.988432] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.991425] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.994501] NET: Registered protocol family 1 [ 0.996835] RPC: Registered named UNIX socket transport module. [ 0.999127] RPC: Registered udp transport module. [ 1.000830] RPC: Registered tcp transport module. [ 1.002595] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.005306] NET: Registered protocol family 44 [ 1.007175] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.009601] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.011761] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.014535] PCI: CLS 0 bytes, default 64 [ 1.016368] Unpacking initramfs... [ 2.446724] debug: unmapping init [mem 0xffff9534fcc54000-0xffff9534fffbffff] [ 2.453100] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.455480] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.460865] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.517544] Initialise system trusted keyrings [ 3.522609] Key type blacklist registered [ 3.527607] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.550163] zbud: loaded [ 3.555498] *** VALIDATE nfs *** [ 3.557127] *** VALIDATE nfs4 *** [ 3.559408] pstore: using deflate compression [ 3.563921] Platform Keyring initialized [ 3.862337] NET: Registered protocol family 38 [ 3.863794] Key type asymmetric registered [ 3.864948] Asymmetric key parser 'x509' registered [ 3.866459] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.868637] io scheduler mq-deadline registered [ 3.869779] io scheduler kyber registered [ 3.870988] io scheduler bfq registered [ 3.872674] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.875505] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.877826] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.880950] ACPI: Power Button [PWRF] [ 3.886626] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.895171] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.919121] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.925407] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.940466] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.967913] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.994315] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.000086] Non-volatile memory driver v1.3 [ 4.001882] Linux agpgart interface v0.103 [ 4.054725] virtio_blk virtio1: [vda] 149784 512-byte logical blocks (76.7 MB/73.1 MiB) [ 4.060767] vda: detected capacity change from 0 to 76689408 [ 4.124814] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.132953] vdb: detected capacity change from 0 to 1073741824 [ 4.180286] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.184105] vdc: detected capacity change from 0 to 2621440000 [ 4.215981] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.224332] vdd: detected capacity change from 0 to 2621440000 [ 4.253839] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.260585] vde: detected capacity change from 0 to 4294967296 [ 4.302393] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.308241] vdf: detected capacity change from 0 to 4294967296 [ 4.332277] libphy: Fixed MDIO Bus: probed [ 4.343108] usbcore: registered new interface driver usbserial_generic [ 4.346662] usbserial: USB Serial support registered for generic [ 4.350723] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.357651] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.359739] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.364591] mousedev: PS/2 mouse device common for all mice [ 4.368775] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.373680] rtc_cmos 00:05: RTC can wake from S4 [ 4.375488] rtc_cmos 00:05: registered as rtc0 [ 4.381662] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.384843] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.390383] intel_pstate: CPU model not supported [ 4.398074] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.401908] hid: raw HID events driver (C) Jiri Kosina [ 4.407718] usbcore: registered new interface driver usbhid [ 4.410686] usbhid: USB HID core driver [ 4.412819] drop_monitor: Initializing network drop monitor service [ 4.416317] Initializing XFRM netlink socket [ 4.418954] NET: Registered protocol family 10 [ 4.422937] Segment Routing with IPv6 [ 4.424844] NET: Registered protocol family 17 [ 4.427441] mpls_gso: MPLS GSO support [ 4.434090] RAS: Correctable Errors collector initialized. [ 4.437667] AVX version of gcm_enc/dec engaged. [ 4.440261] AES CTR mode by8 optimization enabled [ 4.559390] sched_clock: Marking stable (4559361646, 0)->(5403460418, -844098772) [ 4.563561] registered taskstats version 1 [ 4.565629] Loading compiled-in X.509 certificates [ 4.568404] zswap: loaded using pool lzo/zbud [ 4.602930] Key type big_key registered [ 4.626394] Key type encrypted registered [ 4.627992] ima: No TPM chip found, activating TPM-bypass! [ 4.630374] ima: Allocated hash algorithm: sha1 [ 4.632553] ima: No architecture policies found [ 4.634902] evm: Initialising EVM extended attributes: [ 4.637213] evm: security.selinux [ 4.638583] evm: security.ima [ 4.639473] evm: security.capability [ 4.640412] evm: HMAC attrs: 0x1 [ 4.642637] rtc_cmos 00:05: setting system clock to 2026-09-06 03:30:28 UTC (1788665428) [ 4.649538] debug: unmapping init [mem 0xffffffff85803000-0xffffffff859fffff] [ 4.653948] debug: unmapping init [mem 0xffffffff84582000-0xffffffff84858fff] [ 4.662551] Write protecting the kernel read-only data: 28672k [ 4.666885] debug: unmapping init [mem 0xffffffff82c03000-0xffffffff82dfffff] [ 4.670903] debug: unmapping init [mem 0xffffffff83514000-0xffffffff835fffff] [ 4.734099] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.752632] systemd[1]: Detected virtualization kvm. [ 4.755321] systemd[1]: Detected architecture x86-64. [ 4.757148] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.808949] systemd[1]: No hostname configured. [ 4.810771] systemd[1]: Set hostname to . [ 4.826840] random: systemd: uninitialized urandom read (16 bytes read) [ 4.835515] systemd[1]: Initializing machine ID from random generator. [ 4.985250] random: ln: uninitialized urandom read (6 bytes read) [ 5.215243] random: systemd: uninitialized urandom read (16 bytes read) [ 5.218837] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.232757] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.239702] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.592194] device-mapper: uevent: version 1.0.3 [ 6.595521] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 8.206757] random: fast init done Started Hardware RNG Entropy Gatherer Daemon. [ 8.235652] virtio_net virtio0 ens2: renamed from eth0 [ 8.431552] scsi host0: ata_piix [ 8.464565] scsi host1: ata_piix [ 8.465932] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.468553] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.148119] random: crng init done [ 13.150630] random: 7 urandom warning(s) missed due to ratelimiting [ 15.184617] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.427122] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.989603] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.812337] SELinux: Disabled at runtime. [ 19.936050] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.961519] systemd[1]: Detected virtualization kvm. [ 19.968695] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.646414] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.654868] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.664523] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.673890] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.682183] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.742535] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.754764] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Creat[ 22.452664] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 23.142230] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.857879] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.992982] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.411929] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.469178] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 29.574789] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 30.297989] NFS: Registering the id_resolver key type [ 30.303108] Key type id_resolver registered [ 30.310071] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg224-server login: [ 87.034093] hrtimer: interrupt took 12178845 ns [ 92.422871] libcfs: loading out-of-tree module taints kernel. [ 92.461477] Key type ._llcrypt registered [ 92.464951] Key type .llcrypt registered [ 92.559830] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 109.962620] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 111.515376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 111.525602] alg: No test for adler32 (adler32-zlib) [ 112.806256] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 113.630470] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 115.575268] Key type lgssc registered [ 117.123718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 134.874249] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 178.665776] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 190.707085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 190.728043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.929878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 191.963344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 192.035206] Lustre: lustre-MDT0000: new disk, initializing [ 192.096940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.119581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.381382] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.463814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.570643] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 209.603484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 209.607723] Lustre: Skipped 1 previous similar message [ 209.674231] Lustre: lustre-MDT0001: new disk, initializing [ 209.772807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.809026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 209.818410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.224246] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.881869] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 226.671894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.823187] Lustre: lustre-OST0000: new disk, initializing [ 226.825869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 226.832481] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.888926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.337694] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.994495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 233.009162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 233.159463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 248.518968] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.646439] Lustre: lustre-OST0001: new disk, initializing [ 248.650604] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 248.654954] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.716272] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 255.266912] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.516599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 256.524907] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 256.587906] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 266.702734] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 272.907662] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 279.982903] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing check_logdir /tmp/testlogs/ [ 286.080693] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing yml_node [ 290.144505] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 292.772623] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 295.853527] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 297.687954] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 23:35:19 EDT 2026 [ 315.678687] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 317.493336] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.281993] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.295759] Lustre: Skipped 1 previous similar message [ 333.299555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.309532] Lustre: Skipped 2 previous similar messages [ 338.406397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.423243] Lustre: Skipped 3 previous similar messages [ 339.576668] Lustre: server umount lustre-MDT0000 complete [ 344.145280] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665768 with bad export cookie 2226901947964175158 [ 344.153137] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 344.156045] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 348.641694] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.668193] LustreError: 10187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.668631] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.669981] Lustre: Skipped 3 previous similar messages [ 348.709364] LustreError: 10187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 349.676237] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.690177] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 350.751646] Lustre: server umount lustre-MDT0001 complete [ 354.324740] Lustre: server umount lustre-OST0000 complete [ 358.480761] Lustre: server umount lustre-OST0001 complete [ 365.148886] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 374.900713] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 412.764984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.993382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.018618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 413.077581] Lustre: lustre-MDT0000: new disk, initializing [ 413.142439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.158893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 418.034301] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 429.155534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.362707] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 429.473278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 429.475955] Lustre: Skipped 1 previous similar message [ 429.624371] Lustre: lustre-MDT0001: new disk, initializing [ 429.778703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 429.837785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 429.846213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 434.228934] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 444.138776] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 450.294626] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 457.324697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 457.629709] Lustre: lustre-OST0000: new disk, initializing [ 457.635305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 457.650745] Lustre: 17048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 457.775872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.124275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 459.130214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 459.249872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 463.317603] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 471.844489] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 476.492644] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 476.686873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 481.656803] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 481.916972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 484.840499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.842527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 484.866023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 484.869131] Lustre: Skipped 4 previous similar messages [ 489.955378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 489.961655] Lustre: Skipped 1 previous similar message [ 498.143206] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 498.448501] Lustre: server umount lustre-OST0000 complete [ 505.314770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.325565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 505.333365] Lustre: Skipped 2 previous similar messages [ 505.348312] Lustre: Skipped 3 previous similar messages [ 508.518597] Lustre: server umount lustre-MDT0000 complete [ 510.434225] LustreError: 16712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.457670] LustreError: 16712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 512.158212] LustreError: 17047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665936 with bad export cookie 2226901947964176691 [ 512.167617] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 512.183017] LustreError: 17047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 512.665955] Lustre: server umount lustre-MDT0001 complete [ 516.390370] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 23:38:58 (1788665938) [ 523.788772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.663383] LustreError: 19313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.786373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.906678] LustreError: 19314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.417763] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 541.154085] LustreError: 19313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.195628] LustreError: 19313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 542.393220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.749717] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 555.398881] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 561.400970] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 568.185283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 577.169856] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 587.783995] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 596.460133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 596.465830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.479191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 609.249210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 609.512283] Lustre: server umount lustre-OST0000 complete [ 613.859629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 613.878226] Lustre: Skipped 2 previous similar messages [ 619.780750] Lustre: server umount lustre-MDT0000 complete [ 623.801967] LustreError: 19295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666047 with bad export cookie 2226901947964177573 [ 623.816187] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.056489] Lustre: server umount lustre-MDT0001 complete [ 636.876969] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 639.923491] Key type lgssc unregistered [ 640.201302] LNet: 22845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.207230] LNetError: 22845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.258070] LNet: Removed LNI 192.168.202.124@tcp [ 642.173148] Key type .llcrypt unregistered [ 642.178109] Key type ._llcrypt unregistered [ 666.147978] Key type ._llcrypt registered [ 666.150212] Key type .llcrypt registered [ 666.221548] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 667.254523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.270634] alg: No test for adler32 (adler32-zlib) [ 668.313450] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 668.569704] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 670.225220] Key type lgssc registered [ 671.183864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 679.104982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 679.120751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.574963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.699901] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 690.158908] LustreError: 24256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.182801] LustreError: 24256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 694.873499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.154041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 699.525735] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 706.900359] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 713.229578] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 719.618768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 719.779316] LustreError: 26077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 719.896820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 724.923671] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 725.994738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 731.645509] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 749.595318] Lustre: server umount lustre-OST0000 complete [ 756.705611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 756.712089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.718869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 756.720970] Lustre: Skipped 1 previous similar message [ 756.736892] Lustre: Skipped 1 previous similar message [ 761.825498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 761.838935] Lustre: Skipped 1 previous similar message [ 762.970213] Lustre: server umount lustre-MDT0000 complete [ 766.943178] Lustre: 23566:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666174/real 1788666174] req@000000009eb58539 x1875552336113280/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788666190 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 766.948480] LustreError: 24256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 766.964683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.993906] LustreError: 24256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 767.199766] LustreError: 24237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666191 with bad export cookie 8541503520606574807 [ 767.208851] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 767.211620] LustreError: 24237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 767.864745] Lustre: server umount lustre-MDT0001 complete [ 780.140981] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 783.978586] Key type lgssc unregistered [ 784.303056] LNet: 28230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.310185] LNetError: 28230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.329945] LNet: Removed LNI 192.168.202.124@tcp [ 785.471350] Key type .llcrypt unregistered [ 785.476064] Key type ._llcrypt unregistered [ 808.399854] Key type ._llcrypt registered [ 808.402771] Key type .llcrypt registered [ 808.560086] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 809.490538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 809.518095] alg: No test for adler32 (adler32-zlib) [ 810.681675] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 810.933149] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 812.639468] Key type lgssc registered [ 813.713929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 822.186733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 822.232759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.602622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.677595] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 834.022927] LustreError: 29642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 834.040548] LustreError: 29642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 835.417801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 835.739619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 839.482472] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 846.669680] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 852.009445] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 863.597257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 863.876650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.055919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 869.928687] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 877.392551] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 889.831792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 889.844687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.862305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 890.850205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 890.857083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 890.864732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.886645] LustreError: Skipped 1 previous similar message [ 894.801702] Lustre: server umount lustre-OST0000 complete [ 901.103060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.115443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 901.120218] Lustre: Skipped 1 previous similar message [ 901.137762] Lustre: Skipped 2 previous similar messages [ 904.955075] Lustre: server umount lustre-MDT0000 complete [ 906.210338] LustreError: 29641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 906.239675] LustreError: 29641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 908.103171] LustreError: 29622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666331 with bad export cookie 8660823673973845958 [ 908.107962] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 908.115037] LustreError: 29622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 908.460368] Lustre: server umount lustre-MDT0001 complete [ 919.050352] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 921.700295] Key type lgssc unregistered [ 921.905306] LNet: 33519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.915313] LNetError: 33519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.940080] LNet: Removed LNI 192.168.202.124@tcp [ 922.748620] Key type .llcrypt unregistered [ 922.750416] Key type ._llcrypt unregistered [ 944.849753] Key type ._llcrypt registered [ 944.851174] Key type .llcrypt registered [ 944.906266] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 945.807907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 945.899084] alg: No test for adler32 (adler32-zlib) [ 947.057767] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 947.339329] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 948.992169] Key type lgssc registered [ 949.832899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 956.481692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 956.495183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.912946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.951611] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 968.162873] LustreError: 34928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.179303] LustreError: 34928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 970.684649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.977923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 974.986545] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 982.289883] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 989.021752] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 996.587641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 996.872547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.543301] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1004.091771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1009.039360] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1022.990790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1023.010986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.048324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1025.504759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1025.525304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.530620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1025.556927] Lustre: Skipped 1 previous similar message [ 1030.625504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1030.633975] Lustre: Skipped 1 previous similar message [ 1035.744436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1037.279496] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1037.411581] Lustre: server umount lustre-OST0000 complete [ 1042.912284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1042.916844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.925229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1042.929325] Lustre: Skipped 1 previous similar message [ 1045.985660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.201435] Lustre: server umount lustre-MDT0000 complete [ 1050.533222] LustreError: 34908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666474 with bad export cookie 18337531425026153432 [ 1050.544361] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1050.812535] Lustre: server umount lustre-MDT0001 complete [ 1060.487558] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1063.499156] Key type lgssc unregistered [ 1063.692471] LNet: 38483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.697409] LNetError: 38483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.742583] LNet: Removed LNI 192.168.202.124@tcp [ 1065.414851] Key type .llcrypt unregistered [ 1065.416995] Key type ._llcrypt unregistered [ 1088.356321] Key type ._llcrypt registered [ 1088.358310] Key type .llcrypt registered [ 1088.481521] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1089.402319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1089.458488] alg: No test for adler32 (adler32-zlib) [ 1090.709189] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1090.926940] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1092.599261] Key type lgssc registered [ 1093.961846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1101.325299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1101.337388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.721609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1106.651058] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1113.057969] LustreError: 39891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1113.076745] LustreError: 39891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1115.824446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.281727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1120.904752] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1129.138296] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1135.743322] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1143.618837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1143.855620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1147.958546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1150.076734] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1158.563650] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1168.373565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1168.390045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.407181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1170.671068] Lustre: lustre-OST0000: Not available for connect from 192.168.202.24@tcp (stopping) [ 1171.431483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1171.437810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.774770] Lustre: lustre-OST0000: Not available for connect from 192.168.202.24@tcp (stopping) [ 1175.787897] Lustre: Skipped 2 previous similar messages [ 1180.906248] Lustre: lustre-OST0000: Not available for connect from 192.168.202.24@tcp (stopping) [ 1180.923997] Lustre: Skipped 2 previous similar messages [ 1182.175213] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1182.763774] Lustre: server umount lustre-OST0000 complete [ 1207.280783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.282350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.307396] Lustre: Skipped 1 previous similar message [ 1207.330564] Lustre: Skipped 3 previous similar messages [ 1210.583250] Lustre: server umount lustre-MDT0000 complete [ 1211.429253] LustreError: 40618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1211.450420] LustreError: 40618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1214.886083] LustreError: 39872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666638 with bad export cookie 205336360423885244 [ 1214.899791] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1229.281342] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1229.583183] Lustre: server umount lustre-MDT0001 complete [ 1244.089757] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1247.391943] Key type lgssc unregistered [ 1247.754910] LNet: 43601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.763464] LNetError: 43601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.784154] LNet: Removed LNI 192.168.202.124@tcp [ 1248.790967] Key type .llcrypt unregistered [ 1248.798217] Key type ._llcrypt unregistered [ 1278.137867] Key type ._llcrypt registered [ 1278.141532] Key type .llcrypt registered [ 1278.242691] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1280.155494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1280.220550] alg: No test for adler32 (adler32-zlib) [ 1281.409786] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1281.633060] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1283.378932] Key type lgssc registered [ 1284.994721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1293.863301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1293.904456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.568768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1300.357355] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1306.085904] LustreError: 45010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.112514] LustreError: 45010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1310.511777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1310.950461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1316.167251] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1324.951798] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1332.250762] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1339.387363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1339.763638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.804842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1346.968798] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1356.006761] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1366.500918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1366.513394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1366.546868] Lustre: Skipped 2 previous similar messages [ 1366.577229] Lustre: Skipped 2 previous similar messages [ 1369.354208] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.24@tcp (stopping) [ 1371.620980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1371.629244] Lustre: Skipped 2 previous similar messages [ 1374.428872] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.24@tcp (stopping) [ 1379.297556] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1379.462429] Lustre: server umount lustre-MDT0000 complete [ 1379.573403] LustreError: 45005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.865487] LustreError: 45006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.897383] LustreError: 45006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1383.772501] LustreError: 44990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666807 with bad export cookie 2313738996140435524 [ 1383.785795] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1383.796247] LustreError: 44990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1384.669901] LustreError: 45005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1384.718609] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.24@tcp (stopping) [ 1384.722901] Lustre: Skipped 3 previous similar messages [ 1386.979263] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.025445] LustreError: 46000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.078303] LustreError: 46000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1393.185702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1393.193350] Lustre: Skipped 3 previous similar messages [ 1398.241065] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1398.245079] LustreError: 45006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.273707] LustreError: 45006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1398.968605] Lustre: server umount lustre-MDT0001 complete [ 1414.695173] Lustre: server umount lustre-OST0000 complete [ 1436.733641] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1440.206809] Key type lgssc unregistered [ 1440.641375] LNet: 48879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.651528] LNetError: 48879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.702988] LNet: Removed LNI 192.168.202.124@tcp [ 1442.607050] Key type .llcrypt unregistered [ 1442.608660] Key type ._llcrypt unregistered [ 1471.675274] Key type ._llcrypt registered [ 1471.676928] Key type .llcrypt registered [ 1471.787808] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1473.480427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1473.557775] alg: No test for adler32 (adler32-zlib) [ 1474.806524] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1475.034319] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1476.703414] Key type lgssc registered [ 1478.214057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1486.067708] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1486.081660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1501.663440] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1506.783861] LustreError: 50263:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.124@tcp: failed processing log, type 4: rc = -110 [ 1533.472147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.548518] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1548.000385] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1715.917820] Lustre: server umount lustre-OST0000 complete [ 1732.811768] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1735.467418] Key type lgssc unregistered [ 1735.710943] LNet: 51940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.719894] LNetError: 51940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.743969] LNet: Removed LNI 192.168.202.124@tcp [ 1737.514395] Key type .llcrypt unregistered [ 1737.519557] Key type ._llcrypt unregistered [ 1761.539650] Key type ._llcrypt registered [ 1761.542404] Key type .llcrypt registered [ 1761.693648] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1762.963526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1763.107173] alg: No test for adler32 (adler32-zlib) [ 1764.320375] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1764.516317] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1766.191156] Key type lgssc registered [ 1767.423211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1775.162557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1775.185759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.760890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1781.479828] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1786.339248] LustreError: 53352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1786.357219] LustreError: 53352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1791.461371] LustreError: 53353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.510199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.909780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1796.521079] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1804.278413] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1810.819987] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1818.210168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1818.567933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.680821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1824.961655] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1833.824183] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1837.038236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1837.062086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1837.082440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1838.127346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1842.695959] Lustre: server umount lustre-OST0000 complete [ 1848.291864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.303090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1848.314762] Lustre: Skipped 2 previous similar messages [ 1853.409483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1853.428228] Lustre: Skipped 1 previous similar message [ 1858.532323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1858.540775] Lustre: Skipped 1 previous similar message [ 1861.088754] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1861.285252] Lustre: server umount lustre-MDT0000 complete [ 1863.652131] LustreError: 53352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1863.670476] LustreError: 53352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1865.703337] LustreError: 53333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667289 with bad export cookie 5857113983092597135 [ 1865.707631] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1865.717601] LustreError: 53333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1866.167314] Lustre: server umount lustre-MDT0001 complete [ 1878.862980] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1881.585221] Key type lgssc unregistered [ 1881.975570] LNet: 56897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.987075] LNetError: 56897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.001695] LNet: Removed LNI 192.168.202.124@tcp [ 1882.840776] Key type .llcrypt unregistered [ 1882.843084] Key type ._llcrypt unregistered [ 1907.584599] Key type ._llcrypt registered [ 1907.586520] Key type .llcrypt registered [ 1907.701890] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1909.301432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1909.330068] alg: No test for adler32 (adler32-zlib) [ 1910.379745] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1910.556135] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1912.231183] Key type lgssc registered [ 1913.638252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1922.418627] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1922.432548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1937.951426] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1943.071384] LustreError: 58283:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.124@tcp: failed processing log, type 4: rc = -110 [ 1969.760092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1976.553915] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1984.573161] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1991.576510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.237556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.344557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1995.899737] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1997.282591] LustreError: 58309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1997.318253] LustreError: 58309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2002.419197] LustreError: 58308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.457563] LustreError: 58309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.465467] LustreError: 58309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2004.480629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.828375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2009.332575] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2016.384895] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2023.272534] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2025.459839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.461634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2025.468179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2025.480212] Lustre: Skipped 1 previous similar message [ 2030.565265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2030.580641] Lustre: Skipped 2 previous similar messages [ 2031.561162] Lustre: server umount lustre-OST0000 complete [ 2055.905659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.933183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.939886] Lustre: Skipped 1 previous similar message [ 2060.205117] Lustre: server umount lustre-MDT0000 complete [ 2061.284170] LustreError: 59324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.307294] LustreError: 59324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2064.142227] LustreError: 58289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667487 with bad export cookie 7295299921969609656 [ 2064.146327] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2064.164049] LustreError: 58289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2071.185741] Lustre: server umount lustre-MDT0001 complete [ 2088.693114] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2091.654799] Key type lgssc unregistered [ 2091.837646] LNet: 62041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.842808] LNetError: 62041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.902477] LNet: Removed LNI 192.168.202.124@tcp [ 2093.672952] Key type .llcrypt unregistered [ 2093.674583] Key type ._llcrypt unregistered [ 2117.747208] Key type ._llcrypt registered [ 2117.750919] Key type .llcrypt registered [ 2117.894028] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2119.458849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2119.481756] alg: No test for adler32 (adler32-zlib) [ 2120.567206] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 2120.809795] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2122.463271] Key type lgssc registered [ 2123.471070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2130.787246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2130.802947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.183691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.310985] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2142.689918] LustreError: 63452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.704749] LustreError: 63452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2144.571877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.929502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2148.499840] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2155.302789] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2160.566482] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2167.552572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2167.934224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.184912] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2178.167109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2183.741158] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2201.059750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2201.073107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.095313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2204.130154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2204.150530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2204.153261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2204.165537] Lustre: Skipped 1 previous similar message [ 2206.316734] Lustre: server umount lustre-OST0000 complete [ 2211.297078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2211.310753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.323845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2214.369823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2214.385404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.180864] Lustre: server umount lustre-MDT0000 complete [ 2219.488167] LustreError: 65395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2219.509203] LustreError: 65395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2220.409908] LustreError: 63434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667644 with bad export cookie 6384836437349921532 [ 2220.419297] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2220.424933] LustreError: 63434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2220.779296] Lustre: server umount lustre-MDT0001 complete [ 2233.525301] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2235.866740] Key type lgssc unregistered [ 2236.109836] LNet: 67007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2236.124326] LNetError: 67007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2237.160914] LNet: Removed LNI 192.168.202.124@tcp [ 2238.457089] Key type .llcrypt unregistered [ 2238.458788] Key type ._llcrypt unregistered [ 2270.694885] Key type ._llcrypt registered [ 2270.697237] Key type .llcrypt registered [ 2270.786628] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2272.349775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2272.401894] alg: No test for adler32 (adler32-zlib) [ 2273.576949] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 2274.085037] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2275.743678] Key type lgssc registered [ 2276.975306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2289.493488] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 00:08:31 (1788667711) [ 2295.664705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2295.680737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.224400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.250451] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2307.554474] LustreError: 68709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.586916] LustreError: 68709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2310.754677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.224932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2315.689312] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2323.363380] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2329.891724] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2337.499520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2337.856719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2342.958381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2343.755218] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2351.700614] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2362.224600] Lustre: Failing over lustre-MDT0000 [ 2362.338954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2362.351422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.376920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2362.775926] Lustre: server umount lustre-MDT0000 complete [ 2364.664240] LustreError: 68704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.441603] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.445817] LustreError: 68710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.458127] Lustre: Skipped 1 previous similar message [ 2366.491980] LustreError: 68710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2369.768023] LustreError: 71258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.680202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.901109] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.094992] Lustre: *** cfs_fail_loc=135, val=0*** [ 2374.224846] LustreError: 71797:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2374.239098] LustreError: 71797:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.124@tcp: cfg command failed: rc = -2 [ 2374.245195] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2374.245195] [ 2374.254142] LustreError: MGC192.168.202.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2374.275123] LustreError: 71785:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2374.288295] LustreError: 71785:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -2 [ 2374.297749] LustreError: 71785:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-MDT0000 [ 2374.305535] LustreError: 71785:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2374.335826] Lustre: server umount lustre-MDT0000 complete [ 2374.339399] LustreError: 71785:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2374.885952] LustreError: 71258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.908041] LustreError: 71258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2382.591204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.739890] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2382.753514] LustreError: 68701:0:(mgc_request.c:612:do_requeue()) failed processing log: -5 [ 2383.055053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.107769] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2384.989527] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2387.004735] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2388.486791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2388.545497] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2388.615615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2397.156230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2397.172834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.200741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2398.688859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2400.651626] Lustre: server umount lustre-OST0000 complete [ 2408.928225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2408.932501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2408.936459] Lustre: Skipped 1 previous similar message [ 2408.938764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2408.938772] Lustre: Skipped 1 previous similar message [ 2410.670908] Lustre: server umount lustre-MDT0000 complete [ 2414.064145] LustreError: 69437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.091157] LustreError: 69437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2414.816877] LustreError: 68689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667838 with bad export cookie 5384956593531387847 [ 2414.830312] LustreError: 68689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2414.832623] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2415.579681] Lustre: server umount lustre-MDT0001 complete [ 2428.575905] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2432.175601] Key type lgssc unregistered [ 2432.535731] LNet: 73575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2432.541061] LNetError: 73575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.560714] LNet: Removed LNI 192.168.202.124@tcp [ 2433.705052] Key type .llcrypt unregistered [ 2433.708905] Key type ._llcrypt unregistered [ 2460.066277] Key type ._llcrypt registered [ 2460.072380] Key type .llcrypt registered [ 2460.314347] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2461.686174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2461.868312] alg: No test for adler32 (adler32-zlib) [ 2463.278956] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 2463.729996] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2465.487323] Key type lgssc registered [ 2467.201474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2476.962377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2476.988963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.443980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.537758] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2488.808693] LustreError: 74986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.826102] LustreError: 74986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2491.929972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.315818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2496.762789] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2504.675257] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2512.858081] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2520.811114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2521.067193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.160874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2526.785296] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2536.981590] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2543.647294] Lustre: Failing over lustre-MDT0000 [ 2543.972922] Lustre: server umount lustre-MDT0000 complete [ 2544.380193] LustreError: 74982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.687135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.696901] LustreError: 74981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.708825] Lustre: Skipped 1 previous similar message [ 2547.754802] LustreError: 74981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2552.804519] LustreError: 74986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.831667] LustreError: 74986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2554.307456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.400308] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2554.545536] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2554.594642] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.24@tcp (not set up) [ 2554.682598] LustreError: 78072:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2554.691722] LustreError: 78072:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.124@tcp: cfg command failed: rc = -9 [ 2554.703550] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2554.703550] [ 2554.713838] LustreError: MGC192.168.202.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2554.724213] LustreError: 78061:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2554.731897] LustreError: 78061:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -9 [ 2554.746228] LustreError: 78061:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-MDT0000 [ 2554.751399] LustreError: 78061:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2554.772911] Lustre: server umount lustre-MDT0000 complete [ 2554.777636] LustreError: 78061:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2557.924833] LustreError: 74981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.937457] LustreError: 74981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2561.485829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2561.633452] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2561.649095] LustreError: 74979:0:(mgc_request.c:612:do_requeue()) failed processing log: -5 [ 2561.653682] LustreError: 78419:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.124@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000a3216d0d) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2562.031484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.077429] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2564.845942] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2565.762331] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2567.158900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2567.190675] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2567.273091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2577.387113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.391069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2577.416042] Lustre: Skipped 2 previous similar messages [ 2580.365361] Lustre: server umount lustre-OST0000 complete [ 2587.616586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2587.631113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2587.635938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.635946] Lustre: Skipped 1 previous similar message [ 2587.691497] Lustre: Skipped 1 previous similar message [ 2592.738351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2592.748937] Lustre: Skipped 2 previous similar messages [ 2597.858885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.864410] Lustre: Skipped 1 previous similar message [ 2599.391476] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2600.053473] Lustre: server umount lustre-MDT0000 complete [ 2602.977422] LustreError: 74983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.012611] LustreError: 74983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2604.752951] LustreError: 77565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668028 with bad export cookie 597787961048946750 [ 2604.761526] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2604.769117] LustreError: 77565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2605.628652] Lustre: server umount lustre-MDT0001 complete [ 2619.189285] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2621.952830] Key type lgssc unregistered [ 2622.411784] LNet: 79857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2622.420047] LNetError: 79857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.449373] LNet: Removed LNI 192.168.202.124@tcp [ 2623.683539] Key type .llcrypt unregistered [ 2623.689078] Key type ._llcrypt unregistered [ 2643.154726] Key type ._llcrypt registered [ 2643.158576] Key type .llcrypt registered [ 2643.290709] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2643.653443] Key type .llcrypt unregistered [ 2643.657919] Key type ._llcrypt unregistered [ 2675.994560] Key type ._llcrypt registered [ 2676.002115] Key type .llcrypt registered [ 2676.144538] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2677.971505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2678.211057] alg: No test for adler32 (adler32-zlib) [ 2679.619822] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 2679.854836] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2681.521702] Key type lgssc registered [ 2683.509128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2694.869951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2694.907050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2696.742734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.229345] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2706.921169] LustreError: 81754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.958705] LustreError: 81754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2712.059055] LustreError: 81755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2713.469672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.089821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2718.706781] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2727.216965] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2734.786590] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2743.408679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2743.987747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.191204] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2763.087125] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2766.595789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2782.183881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.196314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2782.222593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2782.237574] Lustre: Skipped 1 previous similar message [ 2787.199474] Lustre: server umount lustre-OST0000 complete [ 2792.421500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.426228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2792.443164] Lustre: Skipped 2 previous similar messages [ 2792.452392] Lustre: Skipped 1 previous similar message [ 2797.540816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2797.549420] Lustre: Skipped 1 previous similar message [ 2802.659830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2802.668724] Lustre: Skipped 1 previous similar message [ 2806.240277] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2806.608240] Lustre: server umount lustre-MDT0000 complete [ 2807.781073] LustreError: 82758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2807.807640] LustreError: 82758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2811.654527] LustreError: 83601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668235 with bad export cookie 3278341321772753924 [ 2811.656325] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.660972] LustreError: 83601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2812.351468] Lustre: server umount lustre-MDT0001 complete [ 2828.957966] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2832.651220] Key type lgssc unregistered [ 2832.965496] LNet: 85313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2832.978322] LNetError: 85313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2832.998641] LNet: Removed LNI 192.168.202.124@tcp [ 2834.574807] Key type .llcrypt unregistered [ 2834.578802] Key type ._llcrypt unregistered [ 2867.687051] Key type ._llcrypt registered [ 2867.704156] Key type .llcrypt registered [ 2867.900943] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2868.656498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2868.711576] alg: No test for adler32 (adler32-zlib) [ 2870.144263] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 2870.436033] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2872.215195] Key type lgssc registered [ 2873.726669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2884.265841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2884.274639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.799219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.888869] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2895.332705] LustreError: 86723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.344325] LustreError: 86723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2900.452748] LustreError: 86722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2902.020633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.797880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2908.618140] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2917.363812] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2923.725663] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2932.049600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2932.336676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.584586] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2949.637863] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2955.904430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2966.502028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2966.518553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2966.529386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2966.550265] Lustre: Skipped 1 previous similar message [ 2970.524764] Lustre: server umount lustre-OST0000 complete [ 2976.737774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.743450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2976.751527] Lustre: Skipped 2 previous similar messages [ 2981.740522] Lustre: server umount lustre-MDT0000 complete [ 2981.861215] LustreError: 86722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.896035] LustreError: 86722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2985.873910] LustreError: 86702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668409 with bad export cookie 5618670088651083020 [ 2985.883453] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2985.891471] LustreError: 86702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2986.363466] Lustre: server umount lustre-MDT0001 complete [ 2999.972199] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 3003.333555] Key type lgssc unregistered [ 3003.659554] LNet: 90279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3003.670868] LNetError: 90279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3003.693590] LNet: Removed LNI 192.168.202.124@tcp [ 3004.708127] Key type .llcrypt unregistered [ 3004.714622] Key type ._llcrypt unregistered [ 3032.734544] Key type ._llcrypt registered [ 3032.739126] Key type .llcrypt registered [ 3032.887219] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3034.086696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3034.143587] alg: No test for adler32 (adler32-zlib) [ 3035.191778] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3035.517700] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 3037.263164] Key type lgssc registered [ 3038.435224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3046.992514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3047.022606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.642723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.326699] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3058.147871] LustreError: 91687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3058.166336] LustreError: 91687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3062.972894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.329660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3068.247203] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3076.471139] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3082.516382] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3089.902897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3090.382067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.498230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3096.534844] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3103.652868] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3122.091334] Lustre: server umount lustre-OST0000 complete [ 3129.316887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.322665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3129.330789] Lustre: Skipped 1 previous similar message [ 3129.346847] Lustre: Skipped 1 previous similar message [ 3134.442049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.455330] Lustre: Skipped 1 previous similar message [ 3135.626709] Lustre: server umount lustre-MDT0000 complete [ 3139.296903] LustreError: 91669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668563 with bad export cookie 9791365655450858679 [ 3139.316831] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3149.991768] Lustre: server umount lustre-MDT0001 complete [ 3162.676547] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 3165.313775] Key type lgssc unregistered [ 3165.732832] LNet: 95245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3165.743231] LNetError: 95245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3165.764375] LNet: Removed LNI 192.168.202.124@tcp [ 3166.850296] Key type .llcrypt unregistered [ 3166.856393] Key type ._llcrypt unregistered [ 3194.274152] Key type ._llcrypt registered [ 3194.275475] Key type .llcrypt registered [ 3194.401925] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3195.589916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3195.614878] alg: No test for adler32 (adler32-zlib) [ 3196.757501] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3197.081897] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 3198.768343] Key type lgssc registered [ 3199.993227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3208.532952] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3208.558532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.159364] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3229.279402] LustreError: 96632:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.124@tcp: failed processing log, type 4: rc = -110 [ 3255.967217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3262.830825] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3270.468362] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3275.293770] Lustre: server umount lustre-OST0000 complete [ 3283.625056] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 00:25:05 (1788668705) [ 3291.623224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3292.047405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3297.116911] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3297.251179] LustreError: 98273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3297.260424] LustreError: 98273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3301.353576] LustreError: 98272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3305.475109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3305.801163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3310.492807] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3318.039075] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3323.899665] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3329.816032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3330.091887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.698480] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3336.235573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3342.871734] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3356.642707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3356.643896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3356.644836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3356.655086] Lustre: Skipped 1 previous similar message [ 3356.659848] LustreError: Skipped 1 previous similar message [ 3360.949777] Lustre: server umount lustre-OST0000 complete [ 3366.880390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.897711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3366.900477] Lustre: Skipped 1 previous similar message [ 3370.470925] Lustre: server umount lustre-MDT0000 complete [ 3372.004700] LustreError: 98268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.029492] LustreError: 98268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3373.919567] LustreError: 98254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668797 with bad export cookie 3473001026136731508 [ 3373.922441] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3373.933773] LustreError: 98254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3374.171880] Lustre: server umount lustre-MDT0001 complete [ 3385.294105] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 3387.690896] Key type lgssc unregistered [ 3387.911575] LNet: 101962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3387.918534] LNetError: 101962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3387.935850] LNet: Removed LNI 192.168.202.124@tcp [ 3388.630166] Key type .llcrypt unregistered [ 3388.632550] Key type ._llcrypt unregistered [ 3430.535392] Key type ._llcrypt registered [ 3430.537450] Key type .llcrypt registered [ 3430.639099] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3431.747163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3431.978370] alg: No test for adler32 (adler32-zlib) [ 3432.986724] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3433.199355] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 3434.831204] Key type lgssc registered [ 3435.775083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3442.294615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3442.313624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.673448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3448.065225] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3453.924024] LustreError: 104146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3453.934807] LustreError: 104146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3456.077422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3456.496765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3460.993571] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3468.962097] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3475.078816] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3483.044403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3483.479360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.644644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3490.383178] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3499.374881] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3511.791749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3511.803885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3511.826733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3512.305065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3512.312533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3512.332108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3512.334208] Lustre: Skipped 1 previous similar message [ 3517.029269] Lustre: server umount lustre-OST0000 complete [ 3522.532170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3522.547817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3522.558390] Lustre: Skipped 1 previous similar message [ 3527.344702] Lustre: server umount lustre-MDT0000 complete [ 3527.648295] LustreError: 104146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.670342] LustreError: 104146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3530.959528] LustreError: 104126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668954 with bad export cookie 6756135792027751386 [ 3530.963846] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3530.974834] LustreError: 104126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3531.312597] Lustre: server umount lustre-MDT0001 complete [ 3542.617582] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 3545.123398] Key type lgssc unregistered [ 3545.409901] LNet: 107679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3545.418287] LNetError: 107679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3546.480593] LNet: Removed LNI 192.168.202.124@tcp [ 3547.481784] Key type .llcrypt unregistered [ 3547.483237] Key type ._llcrypt unregistered [ 3564.254484] Key type ._llcrypt registered [ 3564.256634] Key type .llcrypt registered [ 3564.346713] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3565.492842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3565.542130] alg: No test for adler32 (adler32-zlib) [ 3566.540371] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3566.695552] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 3568.327254] Key type lgssc registered [ 3569.345753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3576.568792] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3576.598326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3592.095839] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3597.215334] LustreError: 108919:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.124@tcp: failed processing log, type 4: rc = -110 [ 3623.903223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3631.459902] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3639.094835] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3646.028271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.505366] LustreError: MGC192.168.202.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3646.535688] LustreError: 109942:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3646.550124] LustreError: 109942:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -2 [ 3646.563469] LustreError: 109942:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-MDT0000 [ 3646.711445] Lustre: server umount lustre-MDT0000 complete [ 3646.713705] LustreError: 109942:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3667.423338] Lustre: 108931:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788669075/real 1788669075] req@00000000807366de x1875555374861696/t0(0) o101->MGC192.168.202.124@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1788669091 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3667.449138] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3667.463533] LustreError: 108931:0:(mgc_request.c:612:do_requeue()) failed processing log: -5 [ 3688.080952] Lustre: server umount lustre-OST0000 complete [ 3697.714155] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 3708.277348] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3746.036282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3746.287450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3746.326754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3746.409643] Lustre: lustre-MDT0000: new disk, initializing [ 3746.465771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3746.479630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3750.544290] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3763.508872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3763.615485] Lustre: 112738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3763.662940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3763.666357] Lustre: Skipped 1 previous similar message [ 3763.747991] Lustre: lustre-MDT0001: new disk, initializing [ 3763.857526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3763.896103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3763.909544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3769.389155] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3778.966559] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3785.489641] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3795.815034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3796.518112] Lustre: lustre-OST0000: new disk, initializing [ 3796.527689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3796.541884] Lustre: 114714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3796.741691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3804.989635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3805.001259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3805.129310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3807.901664] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3819.407253] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3824.688342] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3824.862294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3829.780338] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3830.056807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3835.886815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.892921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3835.941142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3838.635530] Lustre: server umount lustre-OST0000 complete [ 3846.113371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.118468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3846.135559] Lustre: Skipped 2 previous similar messages [ 3846.150156] Lustre: Skipped 2 previous similar messages [ 3848.872425] Lustre: server umount lustre-MDT0000 complete [ 3853.708169] LustreError: 112731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669277 with bad export cookie 14193754112671440512 [ 3853.727226] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3853.921423] Lustre: server umount lustre-MDT0001 complete [ 3866.983263] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 00:34:48 (1788669288) [ 3870.719404] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3887.658291] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 3900.384194] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3934.335426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.533122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3934.557567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3934.630136] Lustre: lustre-MDT0000: new disk, initializing [ 3934.687751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.702647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3939.490875] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3950.628094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3950.783402] Lustre: 119618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3950.832598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3950.836573] Lustre: Skipped 1 previous similar message [ 3950.910578] Lustre: lustre-MDT0001: new disk, initializing [ 3950.991831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3951.020593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3951.036724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3955.923873] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3966.523270] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3974.248602] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3981.554910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3981.947402] Lustre: lustre-OST0000: new disk, initializing [ 3981.952226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3981.960908] Lustre: 121582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3982.097262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3988.613108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3988.628765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3988.749969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3989.421327] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4000.951741] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4007.040713] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4007.257159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4012.725484] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4012.965734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4019.686442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4019.704030] LustreError: Skipped 1 previous similar message [ 4019.708730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.725527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4019.765514] Lustre: Skipped 1 previous similar message [ 4021.804187] Lustre: server umount lustre-OST0000 complete [ 4027.878355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4027.887531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4027.914537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4027.926061] Lustre: Skipped 1 previous similar message [ 4035.048543] LustreError: 121893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.064724] LustreError: 121893:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4035.093751] Lustre: server umount lustre-MDT0000 complete [ 4039.665780] LustreError: 119610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669463 with bad export cookie 14193754112671441394 [ 4039.674699] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4039.690496] LustreError: 119610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4040.082362] Lustre: server umount lustre-MDT0001 complete [ 4051.397401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.940822] LustreError: 123802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.094123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.358780] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4057.569092] LustreError: 123802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4057.582891] LustreError: 123802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4061.668029] LustreError: 123803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.789834] LustreError: 123802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4067.710618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4074.039800] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4083.276951] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4091.210172] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4100.140268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4108.310153] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4117.600863] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4129.358898] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4137.440603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4137.451167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.463700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4137.480659] Lustre: Skipped 2 previous similar messages [ 4137.511732] Lustre: Skipped 3 previous similar messages [ 4141.523755] Lustre: server umount lustre-OST0000 complete [ 4147.680257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4147.683210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4147.707229] Lustre: Skipped 1 previous similar message [ 4147.721766] Lustre: Skipped 1 previous similar message [ 4153.780774] Lustre: server umount lustre-MDT0000 complete [ 4157.945841] LustreError: 123802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4157.972692] LustreError: 123802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4158.276205] LustreError: 124538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669582 with bad export cookie 14193754112671442269 [ 4158.293822] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4158.986280] Lustre: server umount lustre-MDT0001 complete [ 4173.104541] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 4176.685543] Key type lgssc unregistered [ 4177.036832] LNet: 127435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4177.049112] LNetError: 127435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4178.088787] LNet: Removed LNI 192.168.202.124@tcp [ 4179.124181] Key type .llcrypt unregistered [ 4179.125811] Key type ._llcrypt unregistered [ 4199.483302] Key type ._llcrypt registered [ 4199.490094] Key type .llcrypt registered [ 4199.606929] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 4216.955780] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 4218.133666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4218.301380] alg: No test for adler32 (adler32-zlib) [ 4219.382582] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 4219.581668] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 4221.239179] Key type lgssc registered [ 4222.349656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4264.236514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4264.282845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.604576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4265.633917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4265.718492] Lustre: lustre-MDT0000: new disk, initializing [ 4265.855250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.882194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4271.659231] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4283.769039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.891345] Lustre: 130587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4283.941969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4283.945635] Lustre: Skipped 1 previous similar message [ 4283.983373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4284.019800] Lustre: lustre-MDT0001: new disk, initializing [ 4284.094940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4284.127072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4284.135139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4289.669771] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4302.193456] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4309.638672] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4319.928030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4320.417363] Lustre: lustre-OST0000: new disk, initializing [ 4320.426691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4320.442962] Lustre: 132557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4320.587517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4329.069560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4329.095382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4329.264014] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4329.544219] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4343.062241] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4349.072730] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4349.348176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4354.813924] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4355.118796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4360.163993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4360.173431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.196970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4360.218858] Lustre: Skipped 1 previous similar message [ 4364.008452] Lustre: server umount lustre-OST0000 complete [ 4370.400776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4370.404739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4370.424634] Lustre: Skipped 1 previous similar message [ 4370.455649] Lustre: Skipped 1 previous similar message [ 4374.418671] Lustre: server umount lustre-MDT0000 complete [ 4379.203571] LustreError: 130581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669803 with bad export cookie 13596497535956591905 [ 4379.219549] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4379.642375] Lustre: server umount lustre-MDT0001 complete [ 4389.625509] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 00:43:31 (1788669811) [ 4397.411644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4397.775438] LustreError: 134970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.793029] LustreError: 134970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4397.841976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4403.066854] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4403.193292] LustreError: 134971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.335327] LustreError: 134970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.389691] LustreError: 134971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.979069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.236680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4420.267214] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4430.604675] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4438.069873] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4445.152668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4445.161230] LustreError: Skipped 1 previous similar message [ 4445.172698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.201139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4447.264983] Lustre: server umount lustre-MDT0000 complete [ 4450.786842] LustreError: 134965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.830846] LustreError: 134965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4452.074534] LustreError: 134950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669875 with bad export cookie 13596497535956592787 [ 4452.075868] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4452.096519] LustreError: 134950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4452.825884] Lustre: server umount lustre-MDT0001 complete [ 4463.504325] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 00:44:45 (1788669885) [ 4471.431553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4487.135755] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4492.255609] LustreError: 137397:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.124@tcp: failed processing log, type 4: rc = -110 [ 4517.986321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4525.231734] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4533.353679] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4536.289855] Lustre: server umount lustre-OST0000 complete [ 4547.790267] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 00:46:09 (1788669969) [ 4556.632502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.162899] LustreError: 138845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4557.232318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4562.678561] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4572.962118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.484394] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4587.143937] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4593.487813] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4600.388441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4600.831918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4600.835692] Lustre: Skipped 1 previous similar message [ 4609.387945] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4618.509477] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4628.451866] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4628.471630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4628.494988] Lustre: Skipped 2 previous similar messages [ 4628.526940] Lustre: Skipped 3 previous similar messages [ 4633.575170] LustreError: 138845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.599989] LustreError: 138845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4633.712391] Lustre: server umount lustre-MDT0000 complete [ 4637.986989] LustreError: 138825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670061 with bad export cookie 13596497535956593452 [ 4637.997379] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4638.004399] LustreError: 138825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4638.469292] Lustre: server umount lustre-MDT0001 complete [ 4653.409295] Lustre: server umount lustre-OST0000 complete [ 4664.640458] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 00:48:06 (1788670086) [ 4671.905187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.407437] LustreError: 142500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4672.531373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.158941] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4686.521861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4692.276931] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4702.436991] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4709.643894] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4716.917048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4718.328364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4724.646499] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4731.924162] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4737.416858] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4737.635828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4742.406275] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4742.588423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4747.238924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4747.239757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4747.259667] Lustre: Skipped 1 previous similar message [ 4747.266173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4747.285653] Lustre: Skipped 1 previous similar message [ 4751.298339] Lustre: server umount lustre-OST0000 complete [ 4757.473954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4757.490797] Lustre: Skipped 1 previous similar message [ 4761.546091] Lustre: server umount lustre-MDT0000 complete [ 4762.599075] LustreError: 142501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.622612] LustreError: 142501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4765.628736] LustreError: 142480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670189 with bad export cookie 13596497535956594523 [ 4765.632229] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4765.642674] LustreError: 142480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4765.967263] Lustre: server umount lustre-MDT0001 complete [ 4775.210572] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 00:49:57 (1788670197) [ 4782.906526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4798.559932] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4803.679610] LustreError: 146659:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.124@tcp: failed processing log, type 4: rc = -110 [ 4829.407848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.417308] Lustre: Skipped 2 previous similar messages [ 4835.976506] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4845.132493] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4854.659763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4855.437311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4860.294816] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4870.950939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4876.424306] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4884.412274] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4891.669507] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4897.300245] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4897.573503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4902.309560] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4902.481385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4906.981720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4906.986433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4906.994732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4906.994739] Lustre: Skipped 2 previous similar messages [ 4907.003740] Lustre: Skipped 2 previous similar messages [ 4910.758766] Lustre: server umount lustre-MDT0000 complete [ 4912.098689] LustreError: 146684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4912.134377] LustreError: 146684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4915.030817] LustreError: 146665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670338 with bad export cookie 13596497535956595545 [ 4915.050238] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4915.055165] LustreError: 146665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4915.317829] Lustre: server umount lustre-MDT0001 complete [ 4929.934606] Lustre: server umount lustre-OST0000 complete [ 4939.843978] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 00:52:42 (1788670362) [ 4947.229930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4962.975893] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4968.096079] LustreError: 150856:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.124@tcp: failed processing log, type 4: rc = -110 [ 5000.681819] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5009.903317] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5017.116181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5018.044708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5023.303856] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5032.925616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5038.814609] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5048.116697] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5054.509090] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5062.407330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5062.600248] Lustre: lustre-OST0001: new disk, initializing [ 5062.604654] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5062.611215] Lustre: 153728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5069.791181] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5071.952378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5071.974509] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5072.158301] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5080.314490] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5084.966627] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5085.131723] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5090.059672] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5090.282889] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5092.860966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.878429] Lustre: Skipped 1 previous similar message [ 5092.895992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5092.902803] Lustre: Skipped 3 previous similar messages [ 5098.623554] Lustre: server umount lustre-OST0000 complete [ 5103.083783] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5117.409415] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5117.703451] Lustre: server umount lustre-OST0001 complete [ 5136.351211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5136.592278] Lustre: server umount lustre-MDT0000 complete [ 5140.150249] LustreError: 150864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670564 with bad export cookie 13596497535956596497 [ 5140.165092] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5140.168410] LustreError: 150864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5140.547702] Lustre: server umount lustre-MDT0001 complete [ 5163.517906] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 00:56:25 (1788670585) [ 5165.098788] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5166.708996] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5168.229882] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 00:56:30 (1788670590) [ 5175.286505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5175.793289] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5175.827064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5175.998260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5176.007688] Lustre: Skipped 6 previous similar messages [ 5180.894335] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5190.559288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.643707] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5190.671624] Lustre: 157292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5195.785755] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5204.606277] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5211.174456] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5218.383066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5218.536118] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5225.043248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5225.078120] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5232.289702] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5237.495857] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5237.815367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5242.712821] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5242.879301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5245.411739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5245.420801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5245.424082] LustreError: Skipped 1 previous similar message [ 5245.424954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5245.424959] Lustre: Skipped 14 previous similar messages [ 5245.466668] Lustre: Skipped 6 previous similar messages [ 5250.530552] Lustre: server umount lustre-OST0000 complete [ 5274.041466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5280.806700] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5285.568030] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.601470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.607354] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5290.430143] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5296.686124] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5297.004300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5303.217264] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5303.354827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5309.375824] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1f1060f800.ost_server_uuid 50 [ 5311.051068] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c1f1060f800.ost_server_uuid in FULL state after 0 sec [ 5311.327556] LustreError: 157300:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.24@tcp) returned error from blocking AST (req@0000000099da9d27 x1875556059853824 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000433b1bb8/0xbcb07678bc2ac06b lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.24@tcp remote: 0x55670135bf4047d6 expref: 5 pid: 157300 timeout: 5411 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5311.357802] LustreError: lustre-MDT0000: A client on nid 192.168.202.24@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5311.364400] LustreError: 157288:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.24@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000433b1bb8/0xbcb07678bc2ac06b lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.24@tcp remote: 0x55670135bf4047d6 expref: 6 pid: 157300 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5316.587293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5316.608561] LustreError: Skipped 2 previous similar messages [ 5321.515225] Lustre: server umount lustre-OST0000 complete [ 5330.949768] Lustre: server umount lustre-MDT0000 complete [ 5331.941718] LustreError: 157300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.969701] LustreError: 157300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 5334.875596] LustreError: 157285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670758 with bad export cookie 13596497535956597680 [ 5334.883030] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5334.890705] LustreError: 157285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5349.343180] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5349.742978] Lustre: server umount lustre-MDT0001 complete [ 5361.049956] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 5363.730158] Key type lgssc unregistered [ 5363.978747] LNet: 163407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5363.988121] LNetError: 163407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5364.005254] LNet: Removed LNI 192.168.202.124@tcp [ 5364.769469] Key type .llcrypt unregistered [ 5364.772206] Key type ._llcrypt unregistered [ 5388.797775] Key type ._llcrypt registered [ 5388.799441] Key type .llcrypt registered [ 5388.883525] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 5389.809830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5389.880020] alg: No test for adler32 (adler32-zlib) [ 5391.038186] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 5391.228762] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 5392.879247] Key type lgssc registered [ 5394.197984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5403.365867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5403.419545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5405.240150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5410.971979] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5415.396902] LustreError: 164820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5415.428452] LustreError: 164820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5420.515155] LustreError: 164819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.248722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5422.823383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5428.616881] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5437.226284] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5444.279644] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5451.845743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5452.155530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5459.688634] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5468.217417] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5475.729938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5485.288933] Lustre: Failing over lustre-MDT0000 [ 5486.052485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5486.057514] LustreError: 164816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5486.067277] Lustre: Skipped 2 previous similar messages [ 5486.086181] LustreError: 164816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5487.629571] Lustre: server umount lustre-MDT0000 complete [ 5491.176331] LustreError: 165769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5491.195070] LustreError: 165769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5496.297732] LustreError: 164820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5496.321281] LustreError: 164820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5500.090475] LustreError: 164800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.24@tcp arrived at 1788670923 with bad export cookie 4649575181189977733 [ 5501.409409] Lustre: 164131:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788670909/real 1788670909] req@00000000cb08dae7 x1875557288540032/t0(0) o400->MGC192.168.202.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788670925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5501.445483] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5506.531237] LustreError: 164816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5506.552610] LustreError: 164816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5507.845489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5511.993653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5512.099267] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5517.292636] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5517.299888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5517.864707] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5520.428504] Lustre: lustre-MDT0000: Denying connection for new client 82620ef4-dcf6-4b54-9899-1bbd44f9643b (at 192.168.202.24@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5529.056198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5529.063747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.073142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5532.652446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5532.663760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5534.873463] Lustre: server umount lustre-OST0000 complete [ 5539.678138] LustreError: 168585:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5539.702031] Lustre: 167934:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5539.718609] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5539.751805] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5539.771982] LustreError: 167934:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000b90a427 x1875557288560640/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5539.796875] LustreError: 167934:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5539.821297] LustreError: 167934:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5539.846571] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5540.265278] Lustre: server umount lustre-MDT0000 complete [ 5542.883265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.907212] LustreError: 164814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5542.939123] LustreError: 164814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5544.795588] LustreError: 164800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670968 with bad export cookie 4649575181189978062 [ 5544.802168] LustreError: 164127:0:(import.c:751:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5544.804450] LustreError: 164800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5544.822203] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5545.328643] Lustre: server umount lustre-MDT0001 complete [ 5558.818343] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 5562.258903] Key type lgssc unregistered [ 5562.630839] LNet: 169336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5562.639060] LNetError: 169336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5562.681557] LNet: Removed LNI 192.168.202.124@tcp [ 5564.184880] Key type .llcrypt unregistered [ 5564.186650] Key type ._llcrypt unregistered [ 5594.297700] Key type ._llcrypt registered [ 5594.300229] Key type .llcrypt registered [ 5594.368111] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 5596.288735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5596.388066] alg: No test for adler32 (adler32-zlib) [ 5597.620780] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 5597.880951] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 5599.656098] Key type lgssc registered [ 5600.872406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5611.335084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5611.364650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5613.136657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5619.074492] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5622.242957] LustreError: 170747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5622.266452] LustreError: 170747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5627.380574] LustreError: 170748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.091844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.355218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5634.493792] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5642.572647] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5648.070280] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5655.570401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5655.964313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5658.053568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5660.908345] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5668.452721] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5678.560313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5678.568551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.582746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5679.584530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5679.594895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5679.599377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5679.623722] Lustre: Skipped 1 previous similar message [ 5682.010068] Lustre: server umount lustre-OST0000 complete [ 5689.826709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5689.842541] Lustre: Skipped 1 previous similar message [ 5689.843431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5689.861449] Lustre: Skipped 1 previous similar message [ 5692.725319] Lustre: server umount lustre-MDT0000 complete [ 5694.951768] LustreError: 171478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.992158] LustreError: 171478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5696.964757] LustreError: 170729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671120 with bad export cookie 17553924808294356341 [ 5696.978507] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5697.288298] Lustre: server umount lustre-MDT0001 complete [ 5708.300200] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 5711.112380] Key type lgssc unregistered [ 5711.531624] LNet: 174284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5711.539228] LNetError: 174284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5711.561443] LNet: Removed LNI 192.168.202.124@tcp [ 5712.383346] Key type .llcrypt unregistered [ 5712.387850] Key type ._llcrypt unregistered [ 5748.561505] Key type ._llcrypt registered [ 5748.563768] Key type .llcrypt registered [ 5748.653799] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 5749.597197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5749.738832] alg: No test for adler32 (adler32-zlib) [ 5750.936692] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 5751.225867] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 5752.936396] Key type lgssc registered [ 5754.488849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5762.925656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5762.945504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5764.409828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.047734] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5774.819229] LustreError: 175983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.849171] LustreError: 175983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5779.945775] LustreError: 175984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5780.868326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.240309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5786.085732] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5793.827138] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5800.315655] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5808.943842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5809.272889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5814.774598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5816.067925] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5823.388535] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5835.232576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5835.240141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5835.257203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5837.284243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5837.288523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5837.300424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5840.929966] Lustre: server umount lustre-OST0000 complete [ 5847.525552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.536940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5847.541260] Lustre: Skipped 1 previous similar message [ 5847.548361] Lustre: Skipped 1 previous similar message [ 5851.636353] Lustre: server umount lustre-MDT0000 complete [ 5852.643418] LustreError: 175984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.687547] LustreError: 175984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5856.052291] LustreError: 175963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671279 with bad export cookie 1911499784966843220 [ 5856.057482] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5856.060582] LustreError: 175963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5856.500225] Lustre: server umount lustre-MDT0001 complete [ 5868.751664] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 5871.705728] Key type lgssc unregistered [ 5872.008098] LNet: 179517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5872.024344] LNetError: 179517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5872.048655] LNet: Removed LNI 192.168.202.124@tcp [ 5873.049200] Key type .llcrypt unregistered [ 5873.055182] Key type ._llcrypt unregistered [ 5898.274366] Key type ._llcrypt registered [ 5898.285462] Key type .llcrypt registered [ 5898.506892] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 5899.921660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5900.028517] alg: No test for adler32 (adler32-zlib) [ 5901.245680] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 5901.531646] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 5903.207194] Key type lgssc registered [ 5904.609773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5918.451832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5918.464167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.922243] Lustre: *** cfs_fail_loc=135, val=0*** [ 5919.981217] LustreError: 181037:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5919.989541] LustreError: 181037:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.124@tcp: cfg command failed: rc = -2 [ 5919.996722] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5919.996722] [ 5920.007547] LustreError: MGC192.168.202.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5920.020690] LustreError: 180998:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5920.033171] LustreError: 180998:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -2 [ 5920.041635] LustreError: 180998:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-MDT0000 [ 5920.049933] LustreError: 180998:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5920.090854] Lustre: server umount lustre-MDT0000 complete [ 5920.096224] LustreError: 180998:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5927.618554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.124265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5933.030147] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5937.652595] LustreError: 181411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5937.689027] LustreError: 181411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5942.756157] LustreError: 181410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5943.096944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5943.473436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5948.070735] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5955.477648] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5960.571334] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5964.259121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5964.282135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5964.297556] Lustre: Skipped 1 previous similar message [ 5968.416731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5968.427456] Lustre: Skipped 1 previous similar message [ 5969.518141] Lustre: server umount lustre-MDT0000 complete [ 5973.491962] LustreError: 182425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5973.518340] LustreError: 182425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5973.868382] LustreError: 181390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671397 with bad export cookie 2371869401019356333 [ 5973.888145] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5974.343238] Lustre: server umount lustre-MDT0001 complete [ 5988.049307] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 5990.565336] Key type lgssc unregistered [ 5990.835764] LNet: 183731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5990.842535] LNetError: 183731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5991.914182] LNet: Removed LNI 192.168.202.124@tcp [ 5992.775707] Key type .llcrypt unregistered [ 5992.777782] Key type ._llcrypt unregistered [ 6017.727176] Key type ._llcrypt registered [ 6017.729528] Key type .llcrypt registered [ 6017.889838] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 6018.191615] Key type .llcrypt unregistered [ 6018.193721] Key type ._llcrypt unregistered [ 6037.956482] Key type ._llcrypt registered [ 6037.960991] Key type .llcrypt registered [ 6038.077099] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 6039.573757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6039.634725] alg: No test for adler32 (adler32-zlib) [ 6040.787488] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 6041.134690] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 6042.825097] Key type lgssc registered [ 6043.978798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6051.965809] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6051.985674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6067.551356] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6072.673492] LustreError: 185857:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.124@tcp: failed processing log, type 4: rc = -110 [ 6099.425626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.017882] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6117.442441] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6125.682126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.148576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6126.216276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6131.170250] LustreError: 185882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6131.196263] LustreError: 185882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6131.635134] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6133.225506] LustreError: 185883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6133.251148] LustreError: 185883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6138.350446] LustreError: 186896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6141.845200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.373644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6147.599074] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6156.474332] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6166.056286] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6169.548465] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6182.368301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6182.373967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6182.391520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6183.398891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6183.406101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6183.422050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6185.055964] Lustre: server umount lustre-OST0000 complete [ 6193.634930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6193.652271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6193.658194] Lustre: Skipped 1 previous similar message [ 6193.658735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6193.658744] Lustre: Skipped 1 previous similar message [ 6198.754617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6198.774233] Lustre: Skipped 2 previous similar messages [ 6203.359176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6203.598158] Lustre: server umount lustre-MDT0000 complete [ 6203.875207] LustreError: 186900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6203.891698] LustreError: 186900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6207.230919] LustreError: 185865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671631 with bad export cookie 9227077092448391227 [ 6207.236993] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6207.242276] LustreError: 185865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6207.537555] Lustre: server umount lustre-MDT0001 complete [ 6219.249669] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 6222.230245] Key type lgssc unregistered [ 6222.480814] LNet: 189777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6222.490173] LNetError: 189777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6222.505430] LNet: Removed LNI 192.168.202.124@tcp [ 6223.587167] Key type .llcrypt unregistered [ 6223.590444] Key type ._llcrypt unregistered [ 6251.744361] Key type ._llcrypt registered [ 6251.747831] Key type .llcrypt registered [ 6251.868942] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 6253.209995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6253.277272] alg: No test for adler32 (adler32-zlib) [ 6254.353949] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 6254.661412] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 6256.328905] Key type lgssc registered [ 6257.741268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6266.186229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6266.200204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6267.784750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6274.047647] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6278.113274] LustreError: 191190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6278.145467] LustreError: 191190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6283.236802] LustreError: 191191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6284.136166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6284.574976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6289.766656] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6299.346138] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6306.661258] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6314.603354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6315.069173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.262673] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6330.867074] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6337.602611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6346.113127] Lustre: Failing over lustre-MDT0000 [ 6346.217040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6346.222788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6346.249732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6346.500794] Lustre: server umount lustre-MDT0000 complete [ 6347.019765] LustreError: 191187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6347.750049] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.093042] LustreError: 191186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6352.127575] LustreError: 191186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6357.221866] LustreError: 191185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6357.241713] LustreError: 191185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6364.193032] Lustre: 190499:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788671771/real 1788671771] req@000000004617b294 x1875558193460864/t0(0) o400->MGC192.168.202.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788671787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6364.231919] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6367.467812] LustreError: 191187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6367.507555] LustreError: 191187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6369.093242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6374.887860] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb59fad62cf1c9f43 [ 6374.892874] Lustre: MGC192.168.202.124@tcp: Connection restored to 0@lo (at 0@lo) [ 6375.218810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6375.301475] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6377.704033] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6380.536667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6380.597968] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6380.666947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6381.118666] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6386.901793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6391.978449] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6406.122126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6406.129704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.142298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6406.170626] Lustre: Skipped 2 previous similar messages [ 6408.974917] Lustre: server umount lustre-OST0000 complete [ 6416.354399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6416.360737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.370698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6416.387302] Lustre: Skipped 1 previous similar message [ 6416.390845] Lustre: Skipped 1 previous similar message [ 6419.123576] Lustre: server umount lustre-MDT0000 complete [ 6421.489814] LustreError: 191190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.515149] LustreError: 191190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6423.338380] LustreError: 191172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671847 with bad export cookie 13087369682055110467 [ 6423.353170] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6423.365377] LustreError: 191172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6423.983672] Lustre: server umount lustre-MDT0001 complete [ 6437.778995] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 6441.544192] Key type lgssc unregistered [ 6441.904533] LNet: 196012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6441.916099] LNetError: 196012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6441.953833] LNet: Removed LNI 192.168.202.124@tcp [ 6443.299762] Key type .llcrypt unregistered [ 6443.306906] Key type ._llcrypt unregistered [ 6475.378794] Key type ._llcrypt registered [ 6475.380278] Key type .llcrypt registered [ 6475.570671] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 6477.219247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6477.313122] alg: No test for adler32 (adler32-zlib) [ 6478.388507] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 6478.665430] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 6480.359274] Key type lgssc registered [ 6481.756630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6493.106214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6493.121043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6494.995504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6500.314185] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6500.321226] LustreError: 197424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6500.344851] LustreError: 197424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6504.446330] LustreError: 197423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6509.538606] LustreError: 197424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6510.246608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6510.760392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6518.337430] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6527.914913] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6533.942921] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6541.483475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6541.871655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6548.708362] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6557.600904] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6564.397642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6566.352524] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6576.422883] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6593.010158] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6600.181414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6600.183954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6600.221104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6600.244094] Lustre: Skipped 1 previous similar message [ 6605.296201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6605.313757] Lustre: Skipped 1 previous similar message [ 6606.044996] Lustre: server umount lustre-OST0000 complete [ 6610.411186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.422223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.434053] Lustre: Skipped 2 previous similar messages [ 6610.452176] Lustre: Skipped 1 previous similar message [ 6615.524993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6615.528224] Lustre: Skipped 1 previous similar message [ 6616.804093] Lustre: server umount lustre-MDT0000 complete [ 6620.641369] LustreError: 197420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6620.667040] LustreError: 197420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6621.163338] LustreError: 197404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672045 with bad export cookie 9386954236021456429 [ 6621.174419] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6621.201571] LustreError: 197404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6621.610784] Lustre: server umount lustre-MDT0001 complete [ 6632.656902] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 6635.562888] Key type lgssc unregistered [ 6635.872752] LNet: 201299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6635.879029] LNetError: 201299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6635.896891] LNet: Removed LNI 192.168.202.124@tcp [ 6636.955332] Key type .llcrypt unregistered [ 6636.958685] Key type ._llcrypt unregistered [ 6658.567490] Key type ._llcrypt registered [ 6658.569968] Key type .llcrypt registered [ 6658.661991] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 6659.828296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6660.004908] alg: No test for adler32 (adler32-zlib) [ 6661.138470] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 6661.402619] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 6663.127167] Key type lgssc registered [ 6664.087690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6670.995277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6671.009288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6672.558225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6677.004864] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6683.106634] LustreError: 202560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.120458] LustreError: 202560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6685.289540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6685.624829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6690.214540] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6698.559231] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6703.925058] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6710.509816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6710.890964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6716.692750] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6718.082865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6724.709339] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6731.302615] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6743.081436] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6753.478195] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6763.601528] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6774.752074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6774.755654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6774.786437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6776.800954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6776.802305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6776.808061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6780.014629] Lustre: server umount lustre-OST0000 complete [ 6787.044628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6787.051356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6787.063898] Lustre: Skipped 1 previous similar message [ 6787.089868] Lustre: Skipped 2 previous similar messages [ 6792.165068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6792.175544] Lustre: Skipped 1 previous similar message [ 6796.258565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6796.268723] Lustre: Skipped 1 previous similar message [ 6798.305513] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6798.896486] Lustre: server umount lustre-MDT0000 complete [ 6801.378263] LustreError: 202560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6801.395872] LustreError: 202560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6803.450184] LustreError: 202542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672227 with bad export cookie 14294341815742943891 [ 6803.459618] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6803.459959] LustreError: 202542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6803.882351] Lustre: server umount lustre-MDT0001 complete [ 6815.432384] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 6818.749748] Key type lgssc unregistered [ 6819.083586] LNet: 207580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6819.088965] LNetError: 207580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6819.105436] LNet: Removed LNI 192.168.202.124@tcp [ 6820.525513] Key type .llcrypt unregistered [ 6820.528326] Key type ._llcrypt unregistered [ 6849.103837] Key type ._llcrypt registered [ 6849.106827] Key type .llcrypt registered [ 6849.242398] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 6850.327708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6850.545083] alg: No test for adler32 (adler32-zlib) [ 6851.565749] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 6851.847711] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 6853.553585] Key type lgssc registered [ 6854.992691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6862.587253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6862.604664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6864.141904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6868.491859] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6873.577777] LustreError: 208993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6873.615956] LustreError: 208993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6878.531199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6878.692701] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6878.898473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6883.774360] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6893.192335] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6899.822898] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6908.954778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6909.577037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6916.975976] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6928.077452] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6934.155623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6949.856273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6949.870356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6949.873873] LustreError: Skipped 1 previous similar message [ 6949.880456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6949.911750] Lustre: Skipped 1 previous similar message [ 6954.994803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6955.001562] Lustre: Skipped 2 previous similar messages [ 6955.395917] Lustre: server umount lustre-OST0000 complete [ 6960.097046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6960.107321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6960.111148] Lustre: Skipped 1 previous similar message [ 6965.216549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6965.222645] Lustre: Skipped 1 previous similar message [ 6965.478604] Lustre: server umount lustre-MDT0000 complete [ 6969.120943] LustreError: 211491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672392 with bad export cookie 6086548601205421683 [ 6969.122175] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6969.130539] LustreError: 211491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6969.582844] Lustre: server umount lustre-MDT0001 complete [ 6983.696299] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 6986.769964] Key type lgssc unregistered [ 6987.050830] LNet: 212700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6987.061555] LNetError: 212700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6988.139104] LNet: Removed LNI 192.168.202.124@tcp [ 6989.081892] Key type .llcrypt unregistered [ 6989.085538] Key type ._llcrypt unregistered [ 7009.557234] Key type ._llcrypt registered [ 7009.561416] Key type .llcrypt registered [ 7009.661409] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7011.043774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7011.079925] alg: No test for adler32 (adler32-zlib) [ 7012.148789] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 7012.346835] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 7013.983343] Key type lgssc registered [ 7015.180264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7022.541422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7022.557277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7023.964355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7028.272494] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7033.313870] LustreError: 213962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7033.338741] LustreError: 213962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7037.169216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7037.523556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7043.214558] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7051.645126] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7057.988978] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7065.519251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7065.827794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7068.957288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7072.553183] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7080.364972] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7097.852333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7097.855883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.869160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7097.876580] Lustre: Skipped 1 previous similar message [ 7097.883264] Lustre: Skipped 1 previous similar message [ 7101.299036] Lustre: server umount lustre-OST0000 complete [ 7108.067529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7108.081690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7108.082088] Lustre: Skipped 1 previous similar message [ 7108.100410] Lustre: Skipped 1 previous similar message [ 7111.927340] Lustre: server umount lustre-MDT0000 complete [ 7113.190264] LustreError: 213961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7113.239970] LustreError: 213961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7116.226336] LustreError: 213942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672540 with bad export cookie 5556298245070807867 [ 7116.230595] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7116.234486] LustreError: 213942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7116.795888] Lustre: server umount lustre-MDT0001 complete [ 7130.045814] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 7132.732954] Key type lgssc unregistered [ 7133.042652] LNet: 217643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7133.051202] LNetError: 217643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7133.065812] LNet: Removed LNI 192.168.202.124@tcp [ 7133.715656] Key type .llcrypt unregistered [ 7133.719224] Key type ._llcrypt unregistered [ 7154.501704] Key type ._llcrypt registered [ 7154.503583] Key type .llcrypt registered [ 7154.604411] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7156.177286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7156.218345] alg: No test for adler32 (adler32-zlib) [ 7157.427555] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 7157.666193] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 7159.335269] Key type lgssc registered [ 7160.858141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7169.440470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7169.479881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7171.136890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7175.839876] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7180.263361] LustreError: 218909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7180.283260] LustreError: 218909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7185.376898] LustreError: 218910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7185.599527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7185.952730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7190.629171] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7199.135935] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7204.882692] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7211.564496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7212.007993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7216.111419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7217.110389] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7225.357378] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7236.123180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7236.192774] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7236.429076] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7243.724682] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7252.716401] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7255.057246] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7267.757306] Lustre: Permanently deactivating lustre-OST0001 [ 7267.796176] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7274.149398] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7282.229644] Lustre: Permanently reactivating lustre-OST0001 [ 7282.268591] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7282.278951] Lustre: Skipped 2 previous similar messages [ 7282.692049] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.716087] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7282.729454] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7282.755427] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7282.849094] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7298.032306] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7298.034876] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.058343] Lustre: Skipped 1 previous similar message [ 7298.070626] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7298.538747] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7301.091262] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7301.111061] Lustre: Skipped 2 previous similar messages [ 7302.865217] Lustre: server umount lustre-OST0001 complete [ 7306.259899] LustreError: 220730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7306.305222] LustreError: 220730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7311.334116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.350348] LustreError: 220731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.356743] Lustre: Skipped 1 previous similar message [ 7311.361312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7311.387780] LustreError: 220731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7311.420470] Lustre: Skipped 1 previous similar message [ 7315.571997] Lustre: server umount lustre-OST0000 complete [ 7321.570230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7321.571591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7321.602254] Lustre: Skipped 2 previous similar messages [ 7321.619416] Lustre: Skipped 1 previous similar message [ 7326.589061] Lustre: server umount lustre-MDT0000 complete [ 7326.706243] LustreError: 218910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.739517] LustreError: 218910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7330.987082] LustreError: 218889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672754 with bad export cookie 13936832472683019677 [ 7330.991324] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7331.004754] LustreError: 218889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7331.516658] Lustre: server umount lustre-MDT0001 complete [ 7346.693788] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 7350.847174] Key type lgssc unregistered [ 7351.311708] LNet: 223980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7351.321639] LNetError: 223980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7351.354326] LNet: Removed LNI 192.168.202.124@tcp [ 7352.407669] Key type .llcrypt unregistered [ 7352.410510] Key type ._llcrypt unregistered [ 7402.129415] Key type ._llcrypt registered [ 7402.130797] Key type .llcrypt registered [ 7402.225300] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7403.394023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7403.607349] alg: No test for adler32 (adler32-zlib) [ 7404.728148] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 7405.041614] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 7406.671179] Key type lgssc registered [ 7407.714565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7415.872519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7415.902891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7417.416244] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7417.458854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7417.728645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7423.025810] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7433.048599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7433.151384] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7433.176842] Lustre: 226184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7433.424260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7438.169712] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7446.699663] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7454.898641] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7463.517356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7463.799376] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7464.107272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7471.935577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7472.361351] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7481.992337] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7485.519942] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7490.665839] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7497.870402] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7507.369182] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7519.979893] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7519.991519] Lustre: Skipped 1 previous similar message [ 7538.588420] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7538.593493] Lustre: Skipped 1 previous similar message [ 7573.119804] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7573.124949] Lustre: Skipped 4 previous similar messages [ 7641.144689] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7641.151414] Lustre: Skipped 7 previous similar messages [ 7666.659050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7666.669121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7666.677289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7666.686218] Lustre: Skipped 1 previous similar message [ 7666.714936] Lustre: Skipped 1 previous similar message [ 7669.329989] Lustre: server umount lustre-OST0000 complete [ 7674.337080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7674.346153] LustreError: Skipped 1 previous similar message [ 7674.349227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7674.367482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7676.896809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7676.914861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.017967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7682.022798] Lustre: Skipped 2 previous similar messages [ 7687.138214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7687.143795] Lustre: Skipped 1 previous similar message [ 7688.160564] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7688.459486] Lustre: server umount lustre-MDT0000 complete [ 7692.967075] LustreError: 226175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673116 with bad export cookie 11966015606155984354 [ 7692.971155] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7692.979494] LustreError: 226175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7693.566379] Lustre: server umount lustre-MDT0001 complete [ 7709.427474] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 7712.350187] Key type lgssc unregistered [ 7712.715656] LNet: 230990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7712.733543] LNetError: 230990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7712.760345] LNet: Removed LNI 192.168.202.124@tcp [ 7713.888075] Key type .llcrypt unregistered [ 7713.891495] Key type ._llcrypt unregistered [ 7745.194909] Key type ._llcrypt registered [ 7745.196738] Key type .llcrypt registered [ 7745.301528] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7746.963681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7747.011966] alg: No test for adler32 (adler32-zlib) [ 7748.118501] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 7748.356989] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 7750.079482] Key type lgssc registered [ 7751.730430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7760.635574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7760.663534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7762.249528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.936368] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7772.641593] LustreError: 232400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.675501] LustreError: 232400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7777.535501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.831951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7783.453631] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7792.445150] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7800.604883] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7808.774904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7809.074398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7817.053353] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7828.206064] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7832.632126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7850.423249] Lustre: server umount lustre-OST0000 complete [ 7858.148661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.160810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7858.175707] Lustre: Skipped 1 previous similar message [ 7858.208524] Lustre: Skipped 1 previous similar message [ 7863.264980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7863.275133] Lustre: Skipped 1 previous similar message [ 7865.461690] Lustre: server umount lustre-MDT0000 complete [ 7868.385038] Lustre: 231711:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788673276/real 1788673276] req@00000000a4bffd69 x1875559760042496/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788673292 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7868.397793] LustreError: 232395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7868.421503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.455582] LustreError: 232395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7869.420157] Lustre: 231710:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788673277/real 1788673277] req@0000000013d8ee83 x1875559760042752/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788673293 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7870.564157] LustreError: 233133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673294 with bad export cookie 10720339567444932457 [ 7870.568894] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7870.587576] LustreError: 233133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7871.113663] Lustre: server umount lustre-MDT0001 complete [ 7884.902540] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 7888.215802] Key type lgssc unregistered [ 7888.577905] LNet: 236091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7888.589741] LNetError: 236091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7888.608779] LNet: Removed LNI 192.168.202.124@tcp [ 7889.965595] Key type .llcrypt unregistered [ 7889.972126] Key type ._llcrypt unregistered [ 7916.951252] Key type ._llcrypt registered [ 7916.955159] Key type .llcrypt registered [ 7917.074923] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7918.533330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7918.611349] alg: No test for adler32 (adler32-zlib) [ 7919.780425] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 7920.083726] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 7921.792225] Key type lgssc registered [ 7923.215512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8194.577504] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 8197.935235] Key type lgssc unregistered [ 8198.298030] LNet: 238114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8198.315050] LNetError: 238114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8198.343521] LNet: Removed LNI 192.168.202.124@tcp [ 8199.234307] Key type .llcrypt unregistered [ 8199.236128] Key type ._llcrypt unregistered [ 8229.901697] Key type ._llcrypt registered [ 8229.910426] Key type .llcrypt registered [ 8230.034314] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 8252.194352] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8253.854942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8253.910887] alg: No test for adler32 (adler32-zlib) [ 8255.331469] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 8255.561531] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 8257.241176] Key type lgssc registered [ 8258.654493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8298.735894] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7998 sec ========== 01:48:40 (1788673720) [ 8301.859573] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:48:42 (1788673722) === [ 8307.028625] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:48:48 (1788673728) === [ 8353.983865] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 8357.894751] Key type lgssc unregistered [ 8358.416681] LNet: 243349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8358.429434] LNetError: 243349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8358.445313] LNet: Removed LNI 192.168.202.124@tcp [ 8359.759760] Key type .llcrypt unregistered [ 8359.761982] Key type ._llcrypt unregistered