[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498596839 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001019] APIC: Switch to symmetric I/O mode setup [ 0.002597] x2apic enabled [ 0.003014] Switched APIC routing to physical x2apic. [ 0.004034] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.010013] LSM: Security Framework initializing [ 0.011079] Yama: becoming mindful. [ 0.012070] SELinux: Initializing. [ 0.014037] *** VALIDATE selinux *** [ 0.023143] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028797] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030139] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031146] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033009] *** VALIDATE tmpfs *** [ 0.035043] *** VALIDATE proc *** [ 0.036373] *** VALIDATE cgroup *** [ 0.037021] *** VALIDATE cgroup2 *** [ 0.039323] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041097] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044045] Spectre V2 : User space: Vulnerable [ 0.045016] Speculative Store Bypass: Vulnerable [ 0.048341] debug: unmapping init [mem 0xffffffffb2059000-0xffffffffb2060fff] [ 0.050247] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051844] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052034] ... version: 2 [ 0.053018] ... bit width: 48 [ 0.054023] ... generic registers: 4 [ 0.055020] ... value mask: 0000ffffffffffff [ 0.056022] ... max period: 00007fffffffffff [ 0.057025] ... fixed-purpose events: 3 [ 0.058033] ... event mask: 000000070000000f [ 0.059450] rcu: Hierarchical SRCU implementation. [ 0.061806] smp: Bringing up secondary CPUs ... [ 0.062709] x86: Booting SMP configuration: [ 0.063041] .... node #0, CPUs: #1 #2 #3 [ 0.067338] smp: Brought up 1 node, 4 CPUs [ 0.069026] smpboot: Max logical packages: 1 [ 0.070020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.217301] node 0 deferred pages initialised in 144ms [ 0.221171] devtmpfs: initialized [ 0.223480] x86/mm: Memory block size: 128MB [ 0.225203] gcov: version magic: 0x41383552 [ 0.228365] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.232154] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.235478] pinctrl core: initialized pinctrl subsystem [ 0.238297] [ 0.238966] ************************************************************* [ 0.241021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.244020] ** ** [ 0.246023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.249022] ** ** [ 0.252022] ** This means that this kernel is built to expose internal ** [ 0.254015] ** IOMMU data structures, which may compromise security on ** [ 0.257021] ** your system. ** [ 0.259017] ** ** [ 0.262024] ** If you see this message and you are not debugging the ** [ 0.265021] ** kernel, report this immediately to your vendor! ** [ 0.267015] ** ** [ 0.269018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272019] ************************************************************* [ 0.274914] NET: Registered protocol family 16 [ 0.276623] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.279161] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.283103] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.287170] cpuidle: using governor menu [ 0.289000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.291355] PCI: Using configuration type 1 for base access [ 0.293158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.305093] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.307040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.312036] cryptd: max_cpu_qlen set to 1000 [ 0.316278] ACPI: Added _OSI(Module Device) [ 0.318023] ACPI: Added _OSI(Processor Device) [ 0.320017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.322017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.326934] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.333468] ACPI: Interpreter enabled [ 0.335108] ACPI: PM: (supports S0 S3 S4 S5) [ 0.337018] ACPI: Using IOAPIC for interrupt routing [ 0.338160] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.342460] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.353787] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.356052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.359023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.362092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.367379] acpiphp: Slot [2] registered [ 0.369131] acpiphp: Slot [5] registered [ 0.371246] acpiphp: Slot [6] registered [ 0.373206] acpiphp: Slot [7] registered [ 0.374181] acpiphp: Slot [8] registered [ 0.376215] acpiphp: Slot [9] registered [ 0.378265] acpiphp: Slot [10] registered [ 0.379212] acpiphp: Slot [3] registered [ 0.381150] acpiphp: Slot [4] registered [ 0.383183] acpiphp: Slot [11] registered [ 0.384167] acpiphp: Slot [12] registered [ 0.386151] acpiphp: Slot [13] registered [ 0.387223] acpiphp: Slot [14] registered [ 0.389147] acpiphp: Slot [15] registered [ 0.391159] acpiphp: Slot [16] registered [ 0.392135] acpiphp: Slot [17] registered [ 0.394146] acpiphp: Slot [18] registered [ 0.396149] acpiphp: Slot [19] registered [ 0.397186] acpiphp: Slot [20] registered [ 0.399147] acpiphp: Slot [21] registered [ 0.401163] acpiphp: Slot [22] registered [ 0.403140] acpiphp: Slot [23] registered [ 0.404142] acpiphp: Slot [24] registered [ 0.406192] acpiphp: Slot [25] registered [ 0.407093] acpiphp: Slot [26] registered [ 0.409095] acpiphp: Slot [27] registered [ 0.410113] acpiphp: Slot [28] registered [ 0.412115] acpiphp: Slot [29] registered [ 0.413124] acpiphp: Slot [30] registered [ 0.415151] acpiphp: Slot [31] registered [ 0.416065] PCI host bridge to bus 0000:00 [ 0.418019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.420026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.423031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.425024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.428028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.432032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.434197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.437610] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.441607] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.451850] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.457047] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.460032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.462031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.466037] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.468742] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.472129] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.475063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.477906] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.484021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.499022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.504924] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.508947] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.516026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.524025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.546026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.577098] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.585020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.597020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.612021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.625050] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.653027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.670027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.690029] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.707368] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.714018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.719020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.745017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.756156] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.762015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.769017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.783022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.793148] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.821039] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.838024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.858032] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.876934] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.879421] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.882412] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.885431] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.887264] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.893327] iommu: Default domain type: Passthrough [ 0.894564] SCSI subsystem initialized [ 0.895218] ACPI: bus type USB registered [ 0.896126] usbcore: registered new interface driver usbfs [ 0.897116] usbcore: registered new interface driver hub [ 0.898175] usbcore: registered new device driver usb [ 0.902188] pps_core: LinuxPPS API ver. 1 registered [ 0.903013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.906070] PTP clock support registered [ 0.908052] EDAC MC: Ver: 3.0.0 [ 0.910155] PCI: Using ACPI for IRQ routing [ 0.911815] NetLabel: Initializing [ 0.915038] NetLabel: domain hash size = 128 [ 0.919031] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.922146] NetLabel: unlabeled traffic allowed by default [ 0.924126] vgaarb: loaded [ 0.926308] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.927016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.931379] clocksource: Switched to clocksource kvm-clock [ 1.048846] VFS: Disk quotas dquot_6.6.0 [ 1.050846] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.053791] *** VALIDATE ramfs *** [ 1.055267] *** VALIDATE hugetlbfs *** [ 1.057228] pnp: PnP ACPI init [ 1.059910] pnp: PnP ACPI: found 6 devices [ 1.083300] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.086990] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.089426] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.091910] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.094683] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.097498] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.100871] NET: Registered protocol family 2 [ 1.106424] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.113288] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.119103] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.126135] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.130805] TCP: Hash tables configured (established 65536 bind 65536) [ 1.134915] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.139405] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.142774] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.146675] NET: Registered protocol family 1 [ 1.150073] RPC: Registered named UNIX socket transport module. [ 1.152692] RPC: Registered udp transport module. [ 1.154761] RPC: Registered tcp transport module. [ 1.156431] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.159056] NET: Registered protocol family 44 [ 1.160442] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.162923] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.165784] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.168836] PCI: CLS 0 bytes, default 64 [ 1.171714] Unpacking initramfs... [ 2.641363] debug: unmapping init [mem 0xffff92e7bcc54000-0xffff92e7bffbffff] [ 2.645611] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.647652] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.651163] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.168696] Initialise system trusted keyrings [ 3.171532] Key type blacklist registered [ 3.173737] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.185192] zbud: loaded [ 3.189090] *** VALIDATE nfs *** [ 3.190598] *** VALIDATE nfs4 *** [ 3.193044] pstore: using deflate compression [ 3.196340] Platform Keyring initialized [ 3.321091] NET: Registered protocol family 38 [ 3.324787] Key type asymmetric registered [ 3.328237] Asymmetric key parser 'x509' registered [ 3.332686] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.339379] io scheduler mq-deadline registered [ 3.343185] io scheduler kyber registered [ 3.346654] io scheduler bfq registered [ 3.350639] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.353894] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.357315] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.360709] ACPI: Power Button [PWRF] [ 3.367300] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.375206] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.395907] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.402562] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.415788] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.443580] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.473077] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.479403] Non-volatile memory driver v1.3 [ 3.481237] Linux agpgart interface v0.103 [ 3.515207] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.518527] vda: detected capacity change from 0 to 75083776 [ 3.539205] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.542353] vdb: detected capacity change from 0 to 1073741824 [ 3.578331] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.582023] vdc: detected capacity change from 0 to 2621440000 [ 3.600907] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.603994] vdd: detected capacity change from 0 to 2621440000 [ 3.621688] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.625671] vde: detected capacity change from 0 to 4294967296 [ 3.642190] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.645287] vdf: detected capacity change from 0 to 4294967296 [ 3.655528] libphy: Fixed MDIO Bus: probed [ 3.670118] usbcore: registered new interface driver usbserial_generic [ 3.679019] usbserial: USB Serial support registered for generic [ 3.681373] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.686771] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.689249] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.691819] mousedev: PS/2 mouse device common for all mice [ 3.695398] rtc_cmos 00:05: RTC can wake from S4 [ 3.698727] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.700105] rtc_cmos 00:05: registered as rtc0 [ 3.705722] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.708203] intel_pstate: CPU model not supported [ 3.710552] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.717304] hid: raw HID events driver (C) Jiri Kosina [ 3.719898] usbcore: registered new interface driver usbhid [ 3.720233] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.722421] usbhid: USB HID core driver [ 3.728778] drop_monitor: Initializing network drop monitor service [ 3.731423] Initializing XFRM netlink socket [ 3.733080] NET: Registered protocol family 10 [ 3.746738] Segment Routing with IPv6 [ 3.750174] NET: Registered protocol family 17 [ 3.752782] mpls_gso: MPLS GSO support [ 3.763505] RAS: Correctable Errors collector initialized. [ 3.765811] AVX version of gcm_enc/dec engaged. [ 3.767502] AES CTR mode by8 optimization enabled [ 3.883610] sched_clock: Marking stable (3883548303, 0)->(4957648276, -1074099973) [ 3.890660] registered taskstats version 1 [ 3.894357] Loading compiled-in X.509 certificates [ 3.896585] zswap: loaded using pool lzo/zbud [ 3.918970] Key type big_key registered [ 3.931346] Key type encrypted registered [ 3.933137] ima: No TPM chip found, activating TPM-bypass! [ 3.936648] ima: Allocated hash algorithm: sha1 [ 3.938925] ima: No architecture policies found [ 3.941242] evm: Initialising EVM extended attributes: [ 3.943698] evm: security.selinux [ 3.945404] evm: security.ima [ 3.947125] evm: security.capability [ 3.948875] evm: HMAC attrs: 0x1 [ 3.951798] rtc_cmos 00:05: setting system clock to 2026-09-05 15:07:17 UTC (1788620837) [ 3.960856] debug: unmapping init [mem 0xffffffffb3003000-0xffffffffb31fffff] [ 3.964979] debug: unmapping init [mem 0xffffffffb1d82000-0xffffffffb2058fff] [ 3.974144] Write protecting the kernel read-only data: 28672k [ 3.977538] debug: unmapping init [mem 0xffffffffb0403000-0xffffffffb05fffff] [ 3.980290] debug: unmapping init [mem 0xffffffffb0d14000-0xffffffffb0dfffff] [ 4.016784] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.024608] systemd[1]: Detected virtualization kvm. [ 4.026710] systemd[1]: Detected architecture x86-64. [ 4.028713] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.054425] systemd[1]: No hostname configured. [ 4.056534] systemd[1]: Set hostname to . [ 4.058552] random: systemd: uninitialized urandom read (16 bytes read) [ 4.062045] systemd[1]: Initializing machine ID from random generator. [ 4.221746] random: systemd: uninitialized urandom read (16 bytes read) [ 4.224470] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.229498] random: systemd: uninitialized urandom read (16 bytes read) [ 4.232779] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.238254] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.937602] device-mapper: uevent: version 1.0.3 [ 4.939838] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.731450] virtio_net virtio0 ens2: renamed from eth0 [ 5.736641] random: fast init done [ 5.765335] scsi host0: ata_piix [ 5.801180] scsi host1: ata_piix [ 5.847231] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.850489] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.652631] dracut-initqueue[577]: RTNETLINK answers: File exists [ 10.456338] random: crng init done [ 10.457792] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.034566] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.250206] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.522718] SELinux: Disabled at runtime. [ 12.589894] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.598372] systemd[1]: Detected virtualization kvm. [ 12.600347] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.134511] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.139231] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.145101] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.151471] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.154859] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.165381] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.173587] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting udev Coldplug [ 13.330603] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS all Devices... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.723600] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.003466] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.071892] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.194655] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.210088] EDAC sbridge: Ver: 1.1.2 [ 16.137840] Key type dns_resolver registered [ 16.446822] NFS: Registering the id_resolver key type [ 16.449202] Key type id_resolver registered [ 16.450751] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg428-server login: [ 64.848466] hrtimer: interrupt took 8239768 ns [ 77.274536] libcfs: loading out-of-tree module taints kernel. [ 77.365072] Key type ._llcrypt registered [ 77.369834] Key type .llcrypt registered [ 77.552825] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 96.796937] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 98.852545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.873246] alg: No test for adler32 (adler32-zlib) [ 100.330667] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 101.667683] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 103.615244] Key type lgssc registered [ 106.238375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 125.330863] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 173.939501] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 188.334422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 188.404066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.729798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.765863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.874788] Lustre: lustre-MDT0000: new disk, initializing [ 190.011036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.051753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.928183] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.078929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.218561] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.258714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 210.273396] Lustre: Skipped 1 previous similar message [ 210.410919] Lustre: lustre-MDT0001: new disk, initializing [ 210.491363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.530324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 210.543869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.365731] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.638079] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 230.755838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.104982] Lustre: lustre-OST0000: new disk, initializing [ 231.108530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 231.121888] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.196913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.626429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 237.641911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 237.762783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 238.129159] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.897257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.039730] Lustre: lustre-OST0001: new disk, initializing [ 254.042830] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 254.048220] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.120704] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 260.766222] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.725432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 262.735562] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 262.802921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 272.442564] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 283.350231] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 291.478604] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing check_logdir /tmp/testlogs/ [ 297.604767] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing yml_node [ 302.241778] Lustre: DEBUG MARKER: Client: 2.17.57.85 [ 304.679639] Lustre: DEBUG MARKER: MDS: 2.17.57.85 [ 307.244866] Lustre: DEBUG MARKER: OSS: 2.17.57.85 [ 308.911542] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 11:12:20 EDT 2026 [ 327.143480] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 328.785871] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 345.056655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.076497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.078130] Lustre: Skipped 1 previous similar message [ 345.112117] Lustre: Skipped 3 previous similar messages [ 350.178386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 355.296933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 355.306886] Lustre: Skipped 6 previous similar messages [ 359.392620] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 359.728940] Lustre: server umount lustre-MDT0000 complete [ 364.438421] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621197 with bad export cookie 16635548701273830865 [ 364.439935] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 364.470382] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 365.555912] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.559603] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.569080] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.570548] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 365.601584] Lustre: Skipped 3 previous similar messages [ 369.568135] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.568169] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.599225] Lustre: Skipped 2 previous similar messages [ 371.444651] Lustre: server umount lustre-MDT0001 complete [ 377.115948] Lustre: server umount lustre-OST0000 complete [ 382.872436] Lustre: server umount lustre-OST0001 complete [ 390.840268] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 401.760365] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 442.023319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.413989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 442.473835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 442.607550] Lustre: lustre-MDT0000: new disk, initializing [ 442.742788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.781845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 447.910482] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 459.173744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.295743] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 459.355939] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 459.360471] Lustre: Skipped 1 previous similar message [ 459.450994] Lustre: lustre-MDT0001: new disk, initializing [ 459.551527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 459.597632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 459.613806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 464.744340] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 475.066818] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 481.633434] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 488.741876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 488.984463] Lustre: lustre-OST0000: new disk, initializing [ 488.987710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 488.992330] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 489.156279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.447621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 490.455856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 490.598710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 494.783562] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 504.776339] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 509.961388] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 510.219620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 515.037592] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 515.343291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 521.201281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 521.203841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.237385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 521.249614] Lustre: Skipped 1 previous similar message [ 521.276983] Lustre: Skipped 1 previous similar message [ 523.755531] Lustre: server umount lustre-OST0000 complete [ 531.430628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 531.432226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.443036] LustreError: Skipped 1 previous similar message [ 535.959984] Lustre: server umount lustre-MDT0000 complete [ 536.551243] LustreError: 15087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.577132] LustreError: 15087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 540.524730] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621374 with bad export cookie 16635548701273832398 [ 540.525565] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.544488] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 540.920721] Lustre: server umount lustre-MDT0001 complete [ 544.770419] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 11:16:15 (1788621375) [ 552.794969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.498221] LustreError: 19311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.619708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 558.382374] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 559.073101] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.893932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.971076] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 581.068860] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 587.419962] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 594.754310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 602.296237] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 610.941601] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 623.073317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 623.076335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.100516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 623.100523] Lustre: Skipped 2 previous similar messages [ 623.147581] Lustre: Skipped 2 previous similar messages [ 627.897649] Lustre: server umount lustre-OST0000 complete [ 633.317412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.345178] Lustre: Skipped 1 previous similar message [ 638.303763] Lustre: server umount lustre-MDT0000 complete [ 638.436785] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.446896] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 641.875972] LustreError: 19291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621475 with bad export cookie 16635548701273833280 [ 641.882838] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.892022] LustreError: 19291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 642.305913] Lustre: server umount lustre-MDT0001 complete [ 652.957573] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 655.862559] Key type lgssc unregistered [ 656.130811] LNet: 22842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.137464] LNetError: 22842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 657.192359] LNet: Removed LNI 192.168.204.128@tcp [ 658.189223] Key type .llcrypt unregistered [ 658.191572] Key type ._llcrypt unregistered [ 680.710688] Key type ._llcrypt registered [ 680.713143] Key type .llcrypt registered [ 680.803196] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 681.986469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 682.082432] alg: No test for adler32 (adler32-zlib) [ 683.291302] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 683.556370] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 685.288725] Key type lgssc registered [ 686.578780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 695.404115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 695.438560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.276980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 702.235898] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 706.548605] LustreError: 24253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.574451] LustreError: 24253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 711.653292] LustreError: 24254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.383400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.912146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 719.945397] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 729.007106] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 734.980941] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 741.172924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 741.503070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 742.541169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 747.766758] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 754.967891] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 770.533025] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.538251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 770.540222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 770.553583] Lustre: Skipped 1 previous similar message [ 775.653954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 775.657760] Lustre: Skipped 1 previous similar message [ 776.221587] Lustre: server umount lustre-OST0000 complete [ 780.256593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 780.266409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 780.292492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 780.299715] Lustre: Skipped 1 previous similar message [ 785.904983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 785.911236] Lustre: Skipped 1 previous similar message [ 786.251940] Lustre: server umount lustre-MDT0000 complete [ 790.282779] LustreError: 24234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621623 with bad export cookie 516896741858189766 [ 790.297087] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 790.307663] LustreError: 24234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 790.863351] Lustre: server umount lustre-MDT0001 complete [ 803.502183] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 806.339237] Key type lgssc unregistered [ 806.725941] LNet: 28229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.741393] LNetError: 28229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.760805] LNet: Removed LNI 192.168.204.128@tcp [ 807.975234] Key type .llcrypt unregistered [ 807.982703] Key type ._llcrypt unregistered [ 832.543368] Key type ._llcrypt registered [ 832.547639] Key type .llcrypt registered [ 832.659706] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 834.359784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 834.443067] alg: No test for adler32 (adler32-zlib) [ 835.608613] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 836.102971] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 837.823313] Key type lgssc registered [ 839.608392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 850.262673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 850.291572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.826045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.873202] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 862.193611] LustreError: 29632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 862.251517] LustreError: 29632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 867.299140] LustreError: 29633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.337031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.831736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 876.111934] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 885.616661] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 892.785862] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 908.505672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 908.977226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 915.600807] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 922.294065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 923.359261] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 931.826589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 931.834785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.847530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 932.834970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 932.835787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 932.843726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.858687] Lustre: Skipped 1 previous similar message [ 936.605843] Lustre: server umount lustre-OST0000 complete [ 942.567884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 942.587380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.613861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 946.793750] Lustre: server umount lustre-MDT0000 complete [ 948.193974] LustreError: 31704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.209289] LustreError: 31704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 950.706670] LustreError: 29616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621784 with bad export cookie 3839416147086370122 [ 950.732231] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 951.173076] Lustre: server umount lustre-MDT0001 complete [ 964.916881] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 967.954340] Key type lgssc unregistered [ 968.316718] LNet: 33512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.327178] LNetError: 33512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.338465] LNet: Removed LNI 192.168.204.128@tcp [ 969.245290] Key type .llcrypt unregistered [ 969.251816] Key type ._llcrypt unregistered [ 996.872734] Key type ._llcrypt registered [ 996.876842] Key type .llcrypt registered [ 996.966285] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 997.872334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 997.974090] alg: No test for adler32 (adler32-zlib) [ 999.059194] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 999.311799] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1000.951149] Key type lgssc registered [ 1002.171975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1010.770592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1010.804938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.318972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.969699] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1022.432503] LustreError: 34926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1022.453528] LustreError: 34926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1026.972045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.301534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1033.880609] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1044.091230] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1050.470204] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1057.864195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.285157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.128329] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1074.304320] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1081.845678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1097.186213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.189559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1097.228395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1097.241028] Lustre: Skipped 1 previous similar message [ 1102.379990] Lustre: server umount lustre-OST0000 complete [ 1107.399074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.438847] Lustre: Skipped 1 previous similar message [ 1107.446287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1107.453466] Lustre: Skipped 1 previous similar message [ 1112.553158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1112.563457] Lustre: Skipped 1 previous similar message [ 1114.023373] Lustre: server umount lustre-MDT0000 complete [ 1116.641717] LustreError: 34922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.675221] LustreError: 34922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1118.862180] LustreError: 34908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621952 with bad export cookie 7169359761752168278 [ 1118.864556] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1118.867918] LustreError: 34908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1119.485484] Lustre: server umount lustre-MDT0001 complete [ 1134.648046] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1138.105900] Key type lgssc unregistered [ 1138.609551] LNet: 38482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.621230] LNetError: 38482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.654511] LNet: Removed LNI 192.168.204.128@tcp [ 1140.141840] Key type .llcrypt unregistered [ 1140.148031] Key type ._llcrypt unregistered [ 1169.526064] Key type ._llcrypt registered [ 1169.532404] Key type .llcrypt registered [ 1169.642955] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1171.440607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1171.457899] alg: No test for adler32 (adler32-zlib) [ 1172.434584] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 1172.621540] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1174.265179] Key type lgssc registered [ 1175.644715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1184.811440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1184.838304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.496129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.214257] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1196.002745] LustreError: 39902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.034615] LustreError: 39902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1201.120209] LustreError: 39901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.532145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.912804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1207.888539] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1215.849872] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1222.890574] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1230.398070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1230.731977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.809296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1237.638334] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1246.522861] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1257.444341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1257.452592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.475672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1258.557290] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 1259.507985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.520410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1263.609247] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 1263.622509] Lustre: Skipped 2 previous similar messages [ 1268.723047] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 1268.732772] Lustre: Skipped 2 previous similar messages [ 1270.240436] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1270.477073] Lustre: server umount lustre-OST0000 complete [ 1295.334206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.334262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1295.342270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1295.352581] Lustre: Skipped 1 previous similar message [ 1295.366556] Lustre: Skipped 2 previous similar messages [ 1299.107293] Lustre: server umount lustre-MDT0000 complete [ 1299.496365] LustreError: 41140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.520283] LustreError: 41140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1304.042595] LustreError: 39882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622137 with bad export cookie 18339971899112890413 [ 1304.050545] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1304.059686] LustreError: 39882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1310.642805] Lustre: server umount lustre-MDT0001 complete [ 1323.139282] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1326.476371] Key type lgssc unregistered [ 1326.948055] LNet: 43609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.973550] LNetError: 43609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.011091] LNet: Removed LNI 192.168.204.128@tcp [ 1328.084961] Key type .llcrypt unregistered [ 1328.086907] Key type ._llcrypt unregistered [ 1356.424505] Key type ._llcrypt registered [ 1356.428580] Key type .llcrypt registered [ 1356.591581] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1358.036337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1358.154243] alg: No test for adler32 (adler32-zlib) [ 1359.291699] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 1359.731083] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1361.455180] Key type lgssc registered [ 1362.970696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1371.413710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1371.435026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.966207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.204552] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1383.395055] LustreError: 45021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.418181] LustreError: 45021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1385.991050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.412854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1391.655261] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1400.156776] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1407.162704] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1415.303798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1415.734508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.789755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1423.006417] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1432.767959] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1439.768283] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 1441.258316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.278053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1441.287195] Lustre: Skipped 2 previous similar messages [ 1441.310277] Lustre: Skipped 2 previous similar messages [ 1444.854793] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 1445.685450] Lustre: server umount lustre-MDT0000 complete [ 1446.385406] LustreError: 45016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.436376] LustreError: 45016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1449.976337] LustreError: 46963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.264277] LustreError: 45003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622284 with bad export cookie 4841377946162490175 [ 1451.270100] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1451.277162] LustreError: 45003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1451.512918] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.530041] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1455.096539] LustreError: 45018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.116976] LustreError: 45018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1456.623085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1456.634346] Lustre: Skipped 1 previous similar message [ 1457.759463] Lustre: server umount lustre-MDT0001 complete [ 1479.506924] Lustre: server umount lustre-OST0000 complete [ 1498.071631] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1501.334779] Key type lgssc unregistered [ 1501.739782] LNet: 48883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.745618] LNetError: 48883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.762738] LNet: Removed LNI 192.168.204.128@tcp [ 1503.045743] Key type .llcrypt unregistered [ 1503.047725] Key type ._llcrypt unregistered [ 1532.093512] Key type ._llcrypt registered [ 1532.096111] Key type .llcrypt registered [ 1532.192964] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1533.653422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1533.684850] alg: No test for adler32 (adler32-zlib) [ 1534.853856] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 1535.207855] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1536.896170] Key type lgssc registered [ 1538.197688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1546.495324] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1546.522423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1562.079593] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1567.200092] LustreError: 50268:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 1594.017191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.838641] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1611.153736] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1778.832387] Lustre: server umount lustre-OST0000 complete [ 1798.164654] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1801.020284] Key type lgssc unregistered [ 1801.334693] LNet: 51941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.348360] LNetError: 51941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.372539] LNet: Removed LNI 192.168.204.128@tcp [ 1802.409614] Key type .llcrypt unregistered [ 1802.411476] Key type ._llcrypt unregistered [ 1828.564419] Key type ._llcrypt registered [ 1828.566857] Key type .llcrypt registered [ 1828.661125] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1830.312072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1830.380953] alg: No test for adler32 (adler32-zlib) [ 1831.482819] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 1831.927862] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1833.615781] Key type lgssc registered [ 1835.309233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1845.545290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1845.587347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1847.117794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.144528] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1856.506687] LustreError: 53353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1856.537129] LustreError: 53353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1861.603342] LustreError: 53352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1864.275993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.591325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1869.934817] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1879.109786] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1886.092743] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1894.388546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1894.905324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.867210] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1909.575922] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1915.360566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1915.368868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.390545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1917.476918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1918.458915] Lustre: server umount lustre-OST0000 complete [ 1926.114365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1926.129419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.146812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.155625] Lustre: Skipped 1 previous similar message [ 1927.659221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.773624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1932.786210] Lustre: Skipped 3 previous similar messages [ 1937.375419] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1937.609107] Lustre: server umount lustre-MDT0000 complete [ 1937.890680] LustreError: 53352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.915890] LustreError: 53352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1941.819105] LustreError: 53331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622775 with bad export cookie 5027976315975787296 [ 1941.824174] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1941.837748] LustreError: 53331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1942.148671] Lustre: server umount lustre-MDT0001 complete [ 1954.886129] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1958.092645] Key type lgssc unregistered [ 1958.450861] LNet: 56884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.463988] LNetError: 56884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.494502] LNet: Removed LNI 192.168.204.128@tcp [ 1959.604807] Key type .llcrypt unregistered [ 1959.609428] Key type ._llcrypt unregistered [ 1986.960119] Key type ._llcrypt registered [ 1986.962737] Key type .llcrypt registered [ 1987.111048] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1988.741723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1988.805106] alg: No test for adler32 (adler32-zlib) [ 1989.840476] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 1990.088916] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1991.767476] Key type lgssc registered [ 1993.204930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2002.446775] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2002.500411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2018.143552] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2023.264651] LustreError: 58264:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 2049.953635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.454683] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2065.235856] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2073.230829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.450734] Lustre: MGS: Not available for connect from 0@lo (not set up) [ 2073.801753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.866015] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2078.829957] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2079.203428] LustreError: 58290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.219923] LustreError: 58290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2084.322436] LustreError: 58289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.346878] LustreError: 59298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.373664] LustreError: 59298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2087.458387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.936377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2092.711988] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2101.382812] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2108.293833] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2113.505492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.507869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2113.549076] Lustre: Skipped 1 previous similar message [ 2113.562194] Lustre: Skipped 1 previous similar message [ 2116.787688] Lustre: server umount lustre-OST0000 complete [ 2139.104575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2139.105124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.117266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.137433] Lustre: Skipped 1 previous similar message [ 2139.152843] Lustre: Skipped 1 previous similar message [ 2144.226444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2144.238916] Lustre: Skipped 1 previous similar message [ 2145.404657] Lustre: server umount lustre-MDT0000 complete [ 2149.261638] LustreError: 60044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622982 with bad export cookie 62596709855299624 [ 2149.265561] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.278545] LustreError: 60044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2155.807895] Lustre: server umount lustre-MDT0001 complete [ 2167.640468] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2170.538533] Key type lgssc unregistered [ 2170.831794] LNet: 62011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.841731] LNetError: 62011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.862863] LNet: Removed LNI 192.168.204.128@tcp [ 2171.852511] Key type .llcrypt unregistered [ 2171.855797] Key type ._llcrypt unregistered [ 2196.481865] Key type ._llcrypt registered [ 2196.484441] Key type .llcrypt registered [ 2196.582382] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2197.620595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2197.794083] alg: No test for adler32 (adler32-zlib) [ 2198.983129] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 2199.237905] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2200.943201] Key type lgssc registered [ 2202.467603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2211.211656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2211.223903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.752744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.798672] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2223.074196] LustreError: 63427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.102314] LustreError: 63427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2226.923148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.504985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2231.888439] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2240.310780] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2246.418760] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2253.520057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2253.868491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2257.988176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2261.040642] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2269.757312] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2287.585800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2287.591646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.597700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2287.605852] Lustre: Skipped 1 previous similar message [ 2287.722931] Lustre: server umount lustre-OST0000 complete [ 2292.706885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.712875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.736314] Lustre: Skipped 1 previous similar message [ 2292.751985] Lustre: Skipped 2 previous similar messages [ 2297.826144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2297.834435] Lustre: Skipped 1 previous similar message [ 2298.667019] Lustre: server umount lustre-MDT0000 complete [ 2302.603091] LustreError: 63407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623136 with bad export cookie 9356838075035593039 [ 2302.617931] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2302.972846] Lustre: server umount lustre-MDT0001 complete [ 2314.378745] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2317.395697] Key type lgssc unregistered [ 2317.698867] LNet: 66980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.711355] LNetError: 66980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.726492] LNet: Removed LNI 192.168.204.128@tcp [ 2318.692240] Key type .llcrypt unregistered [ 2318.698084] Key type ._llcrypt unregistered [ 2345.779297] Key type ._llcrypt registered [ 2345.782681] Key type .llcrypt registered [ 2345.877724] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2346.891546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2347.022153] alg: No test for adler32 (adler32-zlib) [ 2348.133217] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 2348.533187] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2350.263367] Key type lgssc registered [ 2352.313380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2360.836445] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 11:46:32 (1788623192) [ 2367.821791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2367.845313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.451353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2374.915944] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2378.724774] LustreError: 68681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.749797] LustreError: 68681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2383.848736] LustreError: 68682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.539850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.983271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2390.419870] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2399.738519] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2406.821561] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2413.808199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2414.223486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2421.146962] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2429.164621] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2438.842747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2446.439968] Lustre: Failing over lustre-MDT0000 [ 2446.716966] Lustre: server umount lustre-MDT0000 complete [ 2449.376621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.377291] LustreError: 68677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.393796] Lustre: Skipped 2 previous similar messages [ 2449.412472] LustreError: 68677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2451.494680] LustreError: 68677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.566338] LustreError: 68676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.588526] LustreError: 68676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2458.501094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.680102] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2458.858553] Lustre: *** cfs_fail_loc=135, val=0*** [ 2458.945962] LustreError: 71793:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2458.956593] LustreError: 71793:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.128@tcp: cfg command failed: rc = -2 [ 2458.977318] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2458.977318] [ 2458.993322] LustreError: MGC192.168.204.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2459.013448] LustreError: 71781:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2459.024639] LustreError: 71781:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 2459.039211] LustreError: 71781:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2459.044674] LustreError: 71781:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2459.067871] Lustre: server umount lustre-MDT0000 complete [ 2459.070383] LustreError: 71781:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2464.739781] LustreError: 69410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.764416] LustreError: 69410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2466.781101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.992791] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2467.279756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.352668] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2467.718124] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2472.180466] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2472.448981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2472.488525] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2472.540065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2481.121527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2481.135069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.164361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2482.659475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2482.661509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.665993] Lustre: server umount lustre-OST0000 complete [ 2492.897127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2492.910522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.935126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2492.936563] Lustre: Skipped 1 previous similar message [ 2492.960362] Lustre: Skipped 2 previous similar messages [ 2498.025559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.039376] Lustre: Skipped 1 previous similar message [ 2503.139125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2503.150470] Lustre: Skipped 1 previous similar message [ 2505.190878] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2505.507095] Lustre: server umount lustre-MDT0000 complete [ 2508.260709] LustreError: 69410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.280664] LustreError: 69410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2510.288687] LustreError: 68663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623343 with bad export cookie 4174922543022502182 [ 2510.294177] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2510.297259] LustreError: 68663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2510.651434] Lustre: server umount lustre-MDT0001 complete [ 2521.830365] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2524.741298] Key type lgssc unregistered [ 2525.123077] LNet: 73575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2525.131583] LNetError: 73575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2525.156308] LNet: Removed LNI 192.168.204.128@tcp [ 2526.064385] Key type .llcrypt unregistered [ 2526.066847] Key type ._llcrypt unregistered [ 2550.198574] Key type ._llcrypt registered [ 2550.201210] Key type .llcrypt registered [ 2550.397497] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2551.940936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2552.179612] alg: No test for adler32 (adler32-zlib) [ 2553.467190] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 2553.729311] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2555.520416] Key type lgssc registered [ 2557.819494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2568.105260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2568.131166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.804117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.049323] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2579.945026] LustreError: 74991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.978874] LustreError: 74991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2585.061595] LustreError: 74990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.976836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.886295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2593.682893] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2602.499483] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2608.956433] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2615.706993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.343493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.161743] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2632.023562] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2639.879586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2649.171543] Lustre: Failing over lustre-MDT0000 [ 2649.511665] Lustre: server umount lustre-MDT0000 complete [ 2650.090525] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.092242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2650.122237] LustreError: 75996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.149958] LustreError: 75996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2654.232540] LustreError: 75996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.360728] LustreError: 75996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.394051] LustreError: 75996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2661.562376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.777313] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.052614] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2662.271973] LustreError: 78101:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2662.278487] LustreError: 78101:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.128@tcp: cfg command failed: rc = -9 [ 2662.295322] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2662.295322] [ 2662.312670] LustreError: MGC192.168.204.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2662.328383] LustreError: 78090:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2662.351564] LustreError: 78090:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 2662.368136] LustreError: 78090:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2662.389902] LustreError: 78090:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2662.474879] Lustre: server umount lustre-MDT0000 complete [ 2662.483271] LustreError: 78090:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2669.556986] LustreError: 74986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.615461] LustreError: 74986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2671.992975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.155643] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2672.174711] LustreError: 74983:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2672.192333] LustreError: 78449:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.204.128@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000043503fa1) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2672.637502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2672.707511] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2674.681215] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2677.759614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2677.812845] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2677.881757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2678.422835] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2687.998842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.006218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.015405] Lustre: Skipped 3 previous similar messages [ 2688.025570] Lustre: Skipped 1 previous similar message [ 2693.089791] LustreError: 76813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2693.102419] LustreError: 76813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2693.193257] Lustre: server umount lustre-OST0000 complete [ 2698.213228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.215208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2698.220302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2698.247484] Lustre: Skipped 1 previous similar message [ 2703.331620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2703.342539] Lustre: Skipped 2 previous similar messages [ 2703.523092] Lustre: server umount lustre-MDT0000 complete [ 2707.676492] LustreError: 74972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623541 with bad export cookie 11744388183454059189 [ 2707.685209] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2707.688180] LustreError: 74972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2708.052165] Lustre: server umount lustre-MDT0001 complete [ 2721.291543] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2724.293398] Key type lgssc unregistered [ 2724.633825] LNet: 79883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2724.650148] LNetError: 79883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2724.671916] LNet: Removed LNI 192.168.204.128@tcp [ 2725.866838] Key type .llcrypt unregistered [ 2725.870175] Key type ._llcrypt unregistered [ 2740.751543] Key type ._llcrypt registered [ 2740.755731] Key type .llcrypt registered [ 2740.905803] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2741.218365] Key type .llcrypt unregistered [ 2741.222436] Key type ._llcrypt unregistered [ 2766.542203] Key type ._llcrypt registered [ 2766.547411] Key type .llcrypt registered [ 2766.686183] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2767.754529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2767.976539] alg: No test for adler32 (adler32-zlib) [ 2769.093193] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 2769.353141] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2771.097285] Key type lgssc registered [ 2772.250125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2779.994765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2780.045253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2781.571679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.133814] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2791.919793] LustreError: 81777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.939133] LustreError: 81777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2795.175266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.235191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2801.752222] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2811.090406] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2818.304819] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2827.050660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2827.177895] LustreError: 83596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.435401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.887512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2833.909669] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2842.329265] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2853.348759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2853.363235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.388807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2856.433734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2856.448873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2856.473093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.964956] Lustre: server umount lustre-OST0000 complete [ 2862.560862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2862.568804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.589939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.596895] Lustre: Skipped 1 previous similar message [ 2866.657677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2866.658314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.756358] Lustre: server umount lustre-MDT0000 complete [ 2871.777783] LustreError: 81774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.796698] LustreError: 81774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2872.846774] LustreError: 84355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623706 with bad export cookie 13744994041066520523 [ 2872.860043] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2872.871633] LustreError: 84355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2873.284227] Lustre: server umount lustre-MDT0001 complete [ 2886.334705] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2889.658923] Key type lgssc unregistered [ 2889.981624] LNet: 85308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2889.989631] LNetError: 85308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2890.012401] LNet: Removed LNI 192.168.204.128@tcp [ 2890.873243] Key type .llcrypt unregistered [ 2890.875287] Key type ._llcrypt unregistered [ 2912.005668] Key type ._llcrypt registered [ 2912.010704] Key type .llcrypt registered [ 2912.119052] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2913.105823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2913.291327] alg: No test for adler32 (adler32-zlib) [ 2914.352582] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 2914.605384] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2916.295240] Key type lgssc registered [ 2918.102186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2928.248444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2928.281524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.798769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.398136] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2940.385315] LustreError: 86697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.415780] LustreError: 86697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2945.513463] LustreError: 86698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.252714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.742766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2951.948927] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2961.233195] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2968.882742] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2980.400042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2981.336380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2990.415796] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2999.291659] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2999.810039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 3006.945709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3006.954593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.987236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3010.022226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3010.026374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.043414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3015.137534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3015.156980] Lustre: Skipped 2 previous similar messages [ 3020.257827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3020.262217] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 3020.271856] Lustre: Skipped 1 previous similar message [ 3020.726687] Lustre: server umount lustre-OST0000 complete [ 3028.460411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3028.475777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.498865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3030.507876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.147313] Lustre: server umount lustre-MDT0000 complete [ 3035.620088] LustreError: 86697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.682383] LustreError: 86697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3036.770545] LustreError: 86677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623870 with bad export cookie 270210687934778415 [ 3036.782846] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3036.791171] LustreError: 86677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3037.179478] Lustre: server umount lustre-MDT0001 complete [ 3048.374769] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 3051.046621] Key type lgssc unregistered [ 3051.417065] LNet: 90235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3051.426708] LNetError: 90235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3051.470426] LNet: Removed LNI 192.168.204.128@tcp [ 3052.354549] Key type .llcrypt unregistered [ 3052.357608] Key type ._llcrypt unregistered [ 3077.423731] Key type ._llcrypt registered [ 3077.429313] Key type .llcrypt registered [ 3077.601162] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3078.761890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3079.002887] alg: No test for adler32 (adler32-zlib) [ 3080.074805] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 3080.350641] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 3082.033955] Key type lgssc registered [ 3083.196382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3093.512507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3093.531847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.160567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.577785] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3104.738260] LustreError: 91646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.778157] LustreError: 91646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3109.857336] LustreError: 91647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3112.104519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.709374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3118.019386] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3128.400688] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3136.411868] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3145.452122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3146.056364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3153.149869] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3162.395706] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3164.553186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3184.103649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3184.115960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.149038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3185.121469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3185.136113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3185.137127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.149236] Lustre: Skipped 1 previous similar message [ 3186.929920] Lustre: server umount lustre-OST0000 complete [ 3194.850217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3194.861663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.880717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3198.036090] Lustre: server umount lustre-MDT0000 complete [ 3200.486467] LustreError: 92629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3200.511577] LustreError: 92629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3202.145187] LustreError: 91625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624035 with bad export cookie 921929415221021246 [ 3202.148493] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3202.155823] LustreError: 91625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3202.486384] Lustre: server umount lustre-MDT0001 complete [ 3215.242792] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 3218.740064] Key type lgssc unregistered [ 3219.138893] LNet: 95203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3219.149848] LNetError: 95203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3219.192222] LNet: Removed LNI 192.168.204.128@tcp [ 3220.316416] Key type .llcrypt unregistered [ 3220.321095] Key type ._llcrypt unregistered [ 3248.076995] Key type ._llcrypt registered [ 3248.082787] Key type .llcrypt registered [ 3248.200590] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3249.673557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3249.703254] alg: No test for adler32 (adler32-zlib) [ 3250.795861] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 3251.030108] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 3252.695226] Key type lgssc registered [ 3253.845651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3264.018417] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3264.034682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3279.586880] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3284.703459] LustreError: 96599:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 3311.396718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3318.422521] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3328.950967] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3335.563311] Lustre: server umount lustre-OST0000 complete [ 3346.422335] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 12:02:57 (1788624177) [ 3355.277829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3355.764678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3360.959270] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3361.271649] LustreError: 98240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3361.290615] LustreError: 98240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3366.371131] LustreError: 98240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.025243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.579275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3376.593522] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3385.706307] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3392.952775] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3401.691934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3402.232965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3408.967520] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3418.335655] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3425.828944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3441.123489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3441.125567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3441.143124] LustreError: Skipped 1 previous similar message [ 3441.174818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3441.177317] Lustre: Skipped 1 previous similar message [ 3441.213817] Lustre: Skipped 1 previous similar message [ 3446.246080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3446.255935] Lustre: Skipped 1 previous similar message [ 3447.354386] Lustre: server umount lustre-OST0000 complete [ 3453.411713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3453.425940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.444088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3456.481944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3456.483662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.997164] Lustre: server umount lustre-MDT0000 complete [ 3461.600859] LustreError: 100081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.623832] LustreError: 100081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3462.259908] LustreError: 98221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624295 with bad export cookie 14545348853349962579 [ 3462.268200] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3462.275266] LustreError: 98221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3462.739687] Lustre: server umount lustre-MDT0001 complete [ 3473.595591] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 3476.716223] Key type lgssc unregistered [ 3477.081210] LNet: 101955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3477.095712] LNetError: 101955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3477.122231] LNet: Removed LNI 192.168.204.128@tcp [ 3478.023198] Key type .llcrypt unregistered [ 3478.025802] Key type ._llcrypt unregistered [ 3527.416967] Key type ._llcrypt registered [ 3527.421838] Key type .llcrypt registered [ 3527.539530] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3528.754520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3528.803591] alg: No test for adler32 (adler32-zlib) [ 3530.181108] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 3530.467622] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 3532.170796] Key type lgssc registered [ 3533.350062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3541.617466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3541.640389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.240069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.746718] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3552.743566] LustreError: 104140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3552.782230] LustreError: 104140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3557.279959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3557.809048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3562.371462] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3570.474142] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3576.990348] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3584.597897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3585.144543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3588.239407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3591.982761] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3600.229463] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3614.198023] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3614.212226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3614.212694] LustreError: Skipped 1 previous similar message [ 3614.220458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3614.231898] Lustre: Skipped 1 previous similar message [ 3614.272950] Lustre: Skipped 1 previous similar message [ 3619.226937] Lustre: server umount lustre-OST0000 complete [ 3624.420480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3624.427698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.434380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3624.451510] Lustre: Skipped 1 previous similar message [ 3624.468430] Lustre: Skipped 1 previous similar message [ 3629.431873] Lustre: server umount lustre-MDT0000 complete [ 3629.541350] LustreError: 104136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3629.561717] LustreError: 104136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3633.191208] LustreError: 104122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624466 with bad export cookie 14218261366826210491 [ 3633.196770] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3633.203030] LustreError: 104122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3633.634538] Lustre: server umount lustre-MDT0001 complete [ 3646.092656] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 3648.825461] Key type lgssc unregistered [ 3649.170041] LNet: 107685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3649.177940] LNetError: 107685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3649.210743] LNet: Removed LNI 192.168.204.128@tcp [ 3650.163391] Key type .llcrypt unregistered [ 3650.166110] Key type ._llcrypt unregistered [ 3670.385401] Key type ._llcrypt registered [ 3670.387598] Key type .llcrypt registered [ 3670.504677] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3672.044868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3672.082156] alg: No test for adler32 (adler32-zlib) [ 3673.375769] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 3673.744573] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 3675.463190] Key type lgssc registered [ 3676.694096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3685.656287] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3685.681767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3701.279686] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3706.399342] LustreError: 108923:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 3733.090904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.515216] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3750.481220] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3759.606351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.084908] LustreError: MGC192.168.204.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3760.095514] LustreError: 109949:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3760.102127] LustreError: 109949:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 3760.106754] LustreError: 109949:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3760.113452] LustreError: 108929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624593 with bad export cookie 14421788590582051913 [ 3760.129120] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3760.182768] Lustre: server umount lustre-MDT0000 complete [ 3760.186699] LustreError: 109949:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3787.494254] Lustre: server umount lustre-OST0000 complete [ 3800.538363] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3812.789572] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3852.234645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3852.625075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3852.650212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3852.752517] Lustre: lustre-MDT0000: new disk, initializing [ 3852.839819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3852.867550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3859.030464] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3873.106538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.228223] Lustre: 112748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3873.317681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3873.323105] Lustre: Skipped 1 previous similar message [ 3873.433418] Lustre: lustre-MDT0001: new disk, initializing [ 3873.580677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3873.655396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3873.676432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3878.949314] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3889.598276] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3896.342382] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3904.269807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3904.498269] Lustre: lustre-OST0000: new disk, initializing [ 3904.500899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3904.506205] Lustre: 114718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3904.589969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3911.795412] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3914.866992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3914.881334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3914.990902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3921.908616] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3926.296572] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3926.543408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3931.955444] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3932.284925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3935.719317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3935.720739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3935.721450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3935.735349] Lustre: Skipped 1 previous similar message [ 3940.571335] Lustre: server umount lustre-OST0000 complete [ 3945.442417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3945.451265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.465380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3945.469904] Lustre: Skipped 1 previous similar message [ 3951.072636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3951.075252] Lustre: Skipped 2 previous similar messages [ 3956.192028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3956.198320] Lustre: Skipped 1 previous similar message [ 3959.263515] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3959.469770] Lustre: server umount lustre-MDT0000 complete [ 3963.126328] LustreError: 112739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624796 with bad export cookie 14421788590582052018 [ 3963.134418] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3963.148733] LustreError: 112739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3963.663990] Lustre: server umount lustre-MDT0001 complete [ 3973.779753] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 12:13:25 (1788624805) [ 3977.378834] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3993.942486] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4002.211102] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4038.936124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4039.153836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4039.181451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4039.255359] Lustre: lustre-MDT0000: new disk, initializing [ 4039.323577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.348456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4043.762811] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4055.182029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4055.251234] Lustre: 119614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4055.302301] LustreError: 119627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4055.313704] LustreError: 119627:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4055.320094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4055.328018] Lustre: Skipped 1 previous similar message [ 4055.391765] Lustre: lustre-MDT0001: new disk, initializing [ 4055.482419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4055.530586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4055.543796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4060.399734] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4070.780934] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4076.558049] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4082.380326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4082.580591] Lustre: lustre-OST0000: new disk, initializing [ 4082.587857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4082.593782] Lustre: 121584:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4082.682489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4088.706084] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4091.465539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4091.473573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4091.584755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4098.765462] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4103.727584] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4103.924804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4108.199721] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4108.455519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4112.355244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.373728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4112.393902] Lustre: Skipped 1 previous similar message [ 4112.423464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4112.438605] Lustre: Skipped 1 previous similar message [ 4116.664818] Lustre: server umount lustre-OST0000 complete [ 4122.080499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4122.102623] LustreError: Skipped 1 previous similar message [ 4122.113175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4122.137312] Lustre: Skipped 1 previous similar message [ 4122.143463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4135.391115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4135.654674] Lustre: server umount lustre-MDT0000 complete [ 4137.955595] LustreError: 120690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.984892] LustreError: 120690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4139.622211] LustreError: 119608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624973 with bad export cookie 14421788590582052900 [ 4139.627123] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4139.645685] LustreError: 119608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4140.267078] Lustre: server umount lustre-MDT0001 complete [ 4150.045793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4150.618959] LustreError: 123806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.753627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.512245] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4155.872213] LustreError: 123807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4160.996153] LustreError: 123806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4164.913896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4169.869292] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4177.363479] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4183.392806] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4191.922214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4200.032574] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4206.567426] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4216.794697] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4225.507738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.510095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4225.532418] Lustre: Skipped 2 previous similar messages [ 4225.555919] Lustre: Skipped 6 previous similar messages [ 4228.061428] Lustre: server umount lustre-OST0000 complete [ 4235.744158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.761544] Lustre: Skipped 1 previous similar message [ 4239.157928] Lustre: server umount lustre-MDT0000 complete [ 4240.864551] LustreError: 124534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.874315] LustreError: 124534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4242.518942] LustreError: 124541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625076 with bad export cookie 14421788590582053782 [ 4242.531476] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4242.856044] Lustre: server umount lustre-MDT0001 complete [ 4253.117988] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4256.351310] Key type lgssc unregistered [ 4256.623140] LNet: 127436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4256.634292] LNetError: 127436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4256.644409] LNet: Removed LNI 192.168.204.128@tcp [ 4257.443289] Key type .llcrypt unregistered [ 4257.445802] Key type ._llcrypt unregistered [ 4274.906967] Key type ._llcrypt registered [ 4274.913064] Key type .llcrypt registered [ 4275.010328] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4291.352880] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4292.578782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4292.687921] alg: No test for adler32 (adler32-zlib) [ 4293.742222] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 4294.001084] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4295.695161] Key type lgssc registered [ 4297.403668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4334.308629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4334.325302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4335.497546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4335.537130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4335.629951] Lustre: lustre-MDT0000: new disk, initializing [ 4335.710204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4335.730379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4339.299892] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4348.280338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.392987] Lustre: 130598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4348.445712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4348.448908] Lustre: Skipped 1 previous similar message [ 4348.507105] Lustre: lustre-MDT0001: new disk, initializing [ 4348.596608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4348.633831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4348.642731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4352.760996] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4361.451463] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4366.216114] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4373.112238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4373.374218] Lustre: lustre-OST0000: new disk, initializing [ 4373.377985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4373.385193] Lustre: 132566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4373.451102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4379.452583] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4382.764105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4382.774368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4382.876157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4387.604969] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4392.379430] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4392.570899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4396.321760] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4396.496841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4398.569534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4398.570073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.604327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4403.169186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4403.176723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4403.193736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4408.805862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4408.813734] Lustre: Skipped 2 previous similar messages [ 4412.383181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4412.531816] Lustre: server umount lustre-OST0000 complete [ 4419.044436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4419.047295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4419.063507] Lustre: Skipped 1 previous similar message [ 4419.080373] Lustre: Skipped 1 previous similar message [ 4421.972044] Lustre: server umount lustre-MDT0000 complete [ 4425.405158] LustreError: 130591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625258 with bad export cookie 14339802440482349286 [ 4425.419744] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4425.656700] Lustre: server umount lustre-MDT0001 complete [ 4433.557698] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 12:21:05 (1788625265) [ 4440.384061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.863386] LustreError: 134979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4440.876665] LustreError: 134979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4440.922972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4446.016872] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4446.187278] LustreError: 134980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.297063] LustreError: 134979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4456.427097] LustreError: 134980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4456.757965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4457.150485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4462.518248] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4471.669695] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4479.688898] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4482.018585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.035067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.040070] Lustre: Skipped 1 previous similar message [ 4493.287950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4493.302266] Lustre: Skipped 5 previous similar messages [ 4496.351146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4496.623129] Lustre: server umount lustre-MDT0000 complete [ 4498.401511] LustreError: 134976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4498.438028] LustreError: 134976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4500.275049] LustreError: 135714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625333 with bad export cookie 14339802440482350168 [ 4500.278392] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4500.295373] LustreError: 135714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4500.546558] Lustre: server umount lustre-MDT0001 complete [ 4509.396473] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 12:22:20 (1788625340) [ 4516.750877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4532.384720] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4537.503647] LustreError: 137400:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 4563.231322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.837602] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4578.676196] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4580.829728] Lustre: server umount lustre-OST0000 complete [ 4589.735765] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 12:23:41 (1788625421) [ 4596.471513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.033392] LustreError: 138845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4597.094054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4601.841615] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4611.011372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.275508] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4624.338125] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4630.019869] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4637.347029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4637.820677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4637.833329] Lustre: Skipped 1 previous similar message [ 4644.696367] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4652.732616] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4667.873385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4667.893519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4667.909758] Lustre: Skipped 1 previous similar message [ 4667.922551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4673.556689] Lustre: server umount lustre-MDT0000 complete [ 4676.986322] LustreError: 139583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625510 with bad export cookie 14339802440482350833 [ 4676.998933] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4677.089189] LustreError: 140010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4677.091953] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4677.112398] LustreError: 140010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4677.132819] Lustre: Skipped 2 previous similar messages [ 4677.268451] Lustre: server umount lustre-MDT0001 complete [ 4682.981536] Lustre: server umount lustre-OST0000 complete [ 4689.959053] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 12:25:21 (1788625521) [ 4695.668701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4700.330345] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4709.571507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4710.012952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4710.015310] Lustre: Skipped 1 previous similar message [ 4714.948072] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4725.233090] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4731.197431] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4738.007792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4741.423716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4744.089524] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4752.304404] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4757.342671] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4757.600990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4761.607652] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4761.798728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4765.166313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4765.175374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4765.177591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4765.177603] Lustre: Skipped 4 previous similar messages [ 4765.218769] Lustre: Skipped 1 previous similar message [ 4769.843370] Lustre: server umount lustre-OST0000 complete [ 4779.791160] Lustre: server umount lustre-MDT0000 complete [ 4780.513468] LustreError: 142515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4780.530697] LustreError: 142515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4783.610921] LustreError: 142500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625617 with bad export cookie 14339802440482351904 [ 4783.613321] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4783.621783] LustreError: 142500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4784.108261] Lustre: server umount lustre-MDT0001 complete [ 4794.883330] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 12:27:06 (1788625626) [ 4802.542161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4818.209466] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4823.329281] LustreError: 146681:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 4849.000250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4849.005328] Lustre: Skipped 1 previous similar message [ 4855.621764] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4864.155407] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4871.662269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.077444] LustreError: 146705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4872.280269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4877.929822] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4888.103106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4893.557276] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4901.301256] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4907.215142] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4911.539793] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4911.772359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4916.100940] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4916.273808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4919.267679] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4919.276523] Lustre: Skipped 2 previous similar messages [ 4919.277179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4919.280485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4919.297343] Lustre: Skipped 5 previous similar messages [ 4924.376157] Lustre: server umount lustre-MDT0000 complete [ 4928.664063] LustreError: 146688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625762 with bad export cookie 14339802440482352926 [ 4928.667043] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4928.674030] LustreError: 146688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4928.966382] Lustre: server umount lustre-MDT0001 complete [ 4943.328665] Lustre: server umount lustre-OST0000 complete [ 4953.597825] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 12:29:44 (1788625784) [ 4961.216869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4976.991697] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4982.111465] LustreError: 150873:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 5013.216520] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5019.757868] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5025.555415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.009534] LustreError: 150898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.027815] LustreError: 150898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5026.201151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5029.824960] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5037.136102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5041.021688] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5047.913615] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5053.297973] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5059.459527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5059.580525] Lustre: lustre-OST0001: new disk, initializing [ 5059.583959] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5059.590242] Lustre: 153749:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5065.758397] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5067.344250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5067.362791] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5067.494745] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5075.632819] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5080.693495] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5080.903731] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5085.093076] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5085.256711] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5087.715226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5087.726467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.741658] Lustre: Skipped 2 previous similar messages [ 5087.756915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5093.086788] Lustre: server umount lustre-OST0000 complete [ 5111.263209] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5111.445824] Lustre: server umount lustre-OST0001 complete [ 5129.183120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5129.370900] Lustre: server umount lustre-MDT0000 complete [ 5133.164781] LustreError: 153748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625966 with bad export cookie 14339802440482353878 [ 5133.169030] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5133.179128] LustreError: 153748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5133.634789] Lustre: server umount lustre-MDT0001 complete [ 5154.432356] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 12:33:06 (1788625986) [ 5155.892953] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5157.759775] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5159.327212] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 12:33:10 (1788625990) [ 5165.651419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.144182] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5166.201838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5166.428483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.436644] Lustre: Skipped 6 previous similar messages [ 5171.014656] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5179.333587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5179.446862] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5179.486845] Lustre: 157312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5184.619832] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5194.267277] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5200.855865] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5209.647995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5209.987072] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5218.585775] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5219.485360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5227.057492] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5232.340302] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5232.607627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5237.522096] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5237.684555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5240.295611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5240.305067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5240.313879] Lustre: Skipped 5 previous similar messages [ 5240.333407] LustreError: Skipped 2 previous similar messages [ 5254.112130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5254.344963] Lustre: server umount lustre-OST0000 complete [ 5276.943889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5280.505138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.536791] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.539811] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5280.559095] Lustre: Skipped 1 previous similar message [ 5283.846214] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5292.696150] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5299.350652] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5299.566691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5304.900302] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5305.311620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5312.558815] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff907346faf800.ost_server_uuid 50 [ 5314.759720] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff907346faf800.ost_server_uuid in FULL state after 0 sec [ 5315.000550] LustreError: 157317:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.28@tcp) returned error from blocking AST (req@00000000d07a63a2 x1875509381437696 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d12359b5/0xc7013658163afc30 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.28@tcp remote: 0xa3301ec1fd20dd69 expref: 5 pid: 157318 timeout: 5414 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5315.079853] LustreError: lustre-MDT0000: A client on nid 192.168.204.28@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5315.089933] LustreError: 157308:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.204.28@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d12359b5/0xc7013658163afc30 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.28@tcp remote: 0xa3301ec1fd20dd69 expref: 6 pid: 157318 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5321.190622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5326.428285] Lustre: server umount lustre-OST0000 complete [ 5336.306729] Lustre: server umount lustre-MDT0000 complete [ 5337.059357] LustreError: 157322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5337.087114] LustreError: 157322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5339.808192] LustreError: 157304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626173 with bad export cookie 14339802440482355061 [ 5339.815272] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5339.820813] LustreError: 157304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5346.388258] Lustre: server umount lustre-MDT0001 complete [ 5358.351626] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5361.660217] Key type lgssc unregistered [ 5362.023219] LNet: 163434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5362.037195] LNetError: 163434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5362.070691] LNet: Removed LNI 192.168.204.128@tcp [ 5363.103884] Key type .llcrypt unregistered [ 5363.106458] Key type ._llcrypt unregistered [ 5388.141063] Key type ._llcrypt registered [ 5388.143041] Key type .llcrypt registered [ 5388.223749] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5388.990144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5389.074654] alg: No test for adler32 (adler32-zlib) [ 5390.461206] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 5390.838129] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5392.495161] Key type lgssc registered [ 5393.822057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5403.653737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5403.685197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5405.182904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5409.376467] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5415.396045] LustreError: 164845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5415.424598] LustreError: 164845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5417.841843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5418.177870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5422.062800] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5428.906138] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5434.034883] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5441.611531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5442.146439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5448.372700] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5450.370136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5456.235904] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5461.705522] Lustre: Failing over lustre-MDT0000 [ 5462.154930] Lustre: server umount lustre-MDT0000 complete [ 5463.074326] LustreError: 164839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5464.546202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5464.551392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5464.564362] LustreError: 164845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5466.083524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5466.093453] Lustre: Skipped 1 previous similar message [ 5471.203101] LustreError: 164844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5471.215673] LustreError: 164844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5474.556879] LustreError: 164825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.28@tcp arrived at 1788626308 with bad export cookie 3750924394745496446 [ 5474.574554] LustreError: 164825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5476.321706] LustreError: 164845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.344888] LustreError: 164845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5480.279818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5480.398347] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5480.634494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5480.688398] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5484.878716] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5486.054721] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5486.062416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5486.970534] Lustre: lustre-MDT0000: Denying connection for new client 1e6e0f27-2c4c-4736-b5b5-8b4637231b24 (at 192.168.204.28@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5496.291646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5496.293511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5496.313851] Lustre: Skipped 1 previous similar message [ 5496.335984] Lustre: Skipped 1 previous similar message [ 5501.408977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5501.416632] Lustre: Skipped 1 previous similar message [ 5506.532582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5506.545755] Lustre: Skipped 1 previous similar message [ 5509.087188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5509.300542] Lustre: server umount lustre-OST0000 complete [ 5513.814655] LustreError: 168577:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5513.831860] Lustre: 167925:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5513.843625] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5513.895136] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5513.937255] LustreError: 167925:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009776663b x1875510531503360/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5514.024730] LustreError: 167925:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5514.031311] LustreError: 167925:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5514.050671] Lustre: lustre-MDT0000: Recovery over after 0:28, of 2 clients 0 recovered and 2 were evicted. [ 5514.749761] Lustre: server umount lustre-MDT0000 complete [ 5516.769434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5516.807337] LustreError: 164839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5519.941185] LustreError: 164826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626353 with bad export cookie 3750924394745496544 [ 5519.945904] LustreError: 164152:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d2dfa9bc x1875510531504384/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5519.946201] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5519.978598] LustreError: 164826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5526.445493] Lustre: server umount lustre-MDT0001 complete [ 5539.233222] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5542.202699] Key type lgssc unregistered [ 5542.449449] LNet: 169328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5542.466215] LNetError: 169328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5542.488343] LNet: Removed LNI 192.168.204.128@tcp [ 5543.318158] Key type .llcrypt unregistered [ 5543.320555] Key type ._llcrypt unregistered [ 5567.134664] Key type ._llcrypt registered [ 5567.138052] Key type .llcrypt registered [ 5567.273373] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5568.223494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5568.259398] alg: No test for adler32 (adler32-zlib) [ 5569.369856] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 5569.583608] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5571.223192] Key type lgssc registered [ 5572.319172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5580.164754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5580.176916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.688678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5587.213590] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5592.035440] LustreError: 170736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5592.059174] LustreError: 170736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5595.897268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5596.402875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5601.003168] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5609.208213] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5614.669078] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5620.634848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5621.063259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5626.430342] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5627.184114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5633.034661] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5640.597291] Lustre: server umount lustre-OST0000 complete [ 5647.843849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5647.854885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.864097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5652.962113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5652.969434] Lustre: Skipped 2 previous similar messages [ 5654.039831] Lustre: server umount lustre-MDT0000 complete [ 5657.540119] LustreError: 170718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626491 with bad export cookie 6353139825620625309 [ 5657.542033] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5657.547543] LustreError: 170718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5667.929912] Lustre: server umount lustre-MDT0001 complete [ 5677.074647] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5679.193114] Key type lgssc unregistered [ 5679.402655] LNet: 174271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5679.414262] LNetError: 174271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5679.439510] LNet: Removed LNI 192.168.204.128@tcp [ 5680.160314] Key type .llcrypt unregistered [ 5680.163903] Key type ._llcrypt unregistered [ 5708.375200] Key type ._llcrypt registered [ 5708.377937] Key type .llcrypt registered [ 5708.510621] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5709.273569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5709.291908] alg: No test for adler32 (adler32-zlib) [ 5710.390581] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 5710.654055] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5712.351184] Key type lgssc registered [ 5713.419274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5721.515764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5721.538127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.895331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5726.711067] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5733.346115] LustreError: 175970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5733.371781] LustreError: 175970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5734.581417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5735.035923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5739.927871] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5748.606339] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5754.513812] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5761.261059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5761.613679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5767.266231] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5767.792491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5774.405782] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5785.571299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.573208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5785.584346] Lustre: Skipped 1 previous similar message [ 5787.292662] Lustre: server umount lustre-OST0000 complete [ 5791.205743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5791.211207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5791.231589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5791.236415] Lustre: Skipped 1 previous similar message [ 5795.809073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5795.809909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5797.246627] Lustre: server umount lustre-MDT0000 complete [ 5800.935984] LustreError: 175970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.949765] LustreError: 175970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5801.112332] LustreError: 175951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626634 with bad export cookie 9569651195012663087 [ 5801.118223] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5801.124586] LustreError: 175951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5801.438804] Lustre: server umount lustre-MDT0001 complete [ 5811.590683] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5814.141827] Key type lgssc unregistered [ 5814.407560] LNet: 179502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5814.416318] LNetError: 179502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5814.435296] LNet: Removed LNI 192.168.204.128@tcp [ 5815.301654] Key type .llcrypt unregistered [ 5815.303862] Key type ._llcrypt unregistered [ 5837.625464] Key type ._llcrypt registered [ 5837.627702] Key type .llcrypt registered [ 5837.728431] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5838.730153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5838.788910] alg: No test for adler32 (adler32-zlib) [ 5839.844143] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 5840.116931] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5841.823309] Key type lgssc registered [ 5842.983856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5853.240584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5853.266947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.629248] Lustre: *** cfs_fail_loc=135, val=0*** [ 5854.702816] LustreError: 181021:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5854.713978] LustreError: 181021:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.128@tcp: cfg command failed: rc = -2 [ 5854.726593] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5854.726593] [ 5854.739030] LustreError: MGC192.168.204.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5854.750264] LustreError: 180982:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5854.757705] LustreError: 180982:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5854.764968] LustreError: 180982:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5854.769579] LustreError: 180982:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5854.838829] Lustre: server umount lustre-MDT0000 complete [ 5854.842204] LustreError: 180982:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5862.085779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5862.430440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5866.346001] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5871.587378] LustreError: 181394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.639258] LustreError: 181394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5875.101077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5875.451524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5879.956756] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5887.265553] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5893.229375] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5896.161359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5896.169994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5896.194801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5897.185752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5897.191644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.321561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5900.331325] Lustre: Skipped 2 previous similar messages [ 5901.533839] Lustre: server umount lustre-MDT0000 complete [ 5905.303617] LustreError: 181375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626738 with bad export cookie 7502056782568202856 [ 5905.319489] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5905.668301] Lustre: server umount lustre-MDT0001 complete [ 5917.342985] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5920.089698] Key type lgssc unregistered [ 5920.395507] LNet: 183712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5920.416336] LNetError: 183712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5920.436159] LNet: Removed LNI 192.168.204.128@tcp [ 5921.242162] Key type .llcrypt unregistered [ 5921.245757] Key type ._llcrypt unregistered [ 5943.556110] Key type ._llcrypt registered [ 5943.560337] Key type .llcrypt registered [ 5943.725232] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5944.070074] Key type .llcrypt unregistered [ 5944.073371] Key type ._llcrypt unregistered [ 5961.156420] Key type ._llcrypt registered [ 5961.158855] Key type .llcrypt registered [ 5961.271197] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5962.485790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5962.527138] alg: No test for adler32 (adler32-zlib) [ 5963.622997] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 5963.867169] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5965.543117] Key type lgssc registered [ 5966.598092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5974.614783] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5974.677905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5990.431549] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5995.551590] LustreError: 185835:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 6022.239232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6029.810680] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6038.795493] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6046.696264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.201896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6047.249578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6051.673779] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6052.322688] LustreError: 185860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.336318] LustreError: 185860:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6057.449722] LustreError: 186882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.469306] LustreError: 186882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6060.470271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.766298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6065.956302] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6074.666637] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6080.508790] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6083.186110] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6096.876093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6096.882089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6096.892323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6096.912549] Lustre: Skipped 1 previous similar message [ 6100.499536] Lustre: server umount lustre-OST0000 complete [ 6107.108643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6107.112194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6107.140029] Lustre: Skipped 1 previous similar message [ 6107.151474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6107.163206] Lustre: Skipped 1 previous similar message [ 6112.224878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6112.241024] Lustre: Skipped 1 previous similar message [ 6117.350196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6117.356884] Lustre: Skipped 1 previous similar message [ 6119.391178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6119.626359] Lustre: server umount lustre-MDT0000 complete [ 6122.477240] LustreError: 188930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.502850] LustreError: 188930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6123.090604] LustreError: 185841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626956 with bad export cookie 9469657603433315636 [ 6123.096356] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6123.099862] LustreError: 185841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6123.351928] Lustre: server umount lustre-MDT0001 complete [ 6133.731464] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 6136.205176] Key type lgssc unregistered [ 6136.451544] LNet: 189812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6136.458537] LNetError: 189812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6136.477772] LNet: Removed LNI 192.168.204.128@tcp [ 6137.317984] Key type .llcrypt unregistered [ 6137.320340] Key type ._llcrypt unregistered [ 6157.448141] Key type ._llcrypt registered [ 6157.452984] Key type .llcrypt registered [ 6157.527689] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6158.240697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6158.256394] alg: No test for adler32 (adler32-zlib) [ 6159.204339] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 6159.391845] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 6161.048325] Key type lgssc registered [ 6161.927291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6168.702576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6168.733601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6170.274313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6174.829186] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6179.810293] LustreError: 191200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.820700] LustreError: 191200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6182.945963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6183.249267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6187.334226] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6194.428653] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6200.218670] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6206.819229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6207.091262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6212.873061] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6215.280918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6220.440282] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6225.408372] Lustre: Failing over lustre-MDT0000 [ 6225.695340] Lustre: server umount lustre-MDT0000 complete [ 6225.898082] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6225.917845] LustreError: 191195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6225.953285] LustreError: 191195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6227.479378] LustreError: 191194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6231.018313] LustreError: 191199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6231.035811] LustreError: 191199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6236.131834] LustreError: 191194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6236.147359] LustreError: 191194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6244.222623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6244.384437] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6244.583809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6244.626323] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6247.923328] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6248.635986] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6249.968332] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6250.004927] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6250.057439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6252.823031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6262.431923] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6275.041268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6275.042732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6275.058647] LustreError: Skipped 1 previous similar message [ 6275.061366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6275.072746] Lustre: Skipped 3 previous similar messages [ 6275.288281] Lustre: server umount lustre-OST0000 complete [ 6280.674667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6280.688449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6280.717198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6280.722785] Lustre: Skipped 2 previous similar messages [ 6285.441489] Lustre: server umount lustre-MDT0000 complete [ 6285.797386] LustreError: 191200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6285.819576] LustreError: 191200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6289.263322] LustreError: 194778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627122 with bad export cookie 7351202634136507341 [ 6289.263863] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6289.273962] LustreError: 194778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6289.593392] Lustre: server umount lustre-MDT0001 complete [ 6300.940534] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 6304.405629] Key type lgssc unregistered [ 6304.755225] LNet: 196133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6304.767289] LNetError: 196133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6304.803343] LNet: Removed LNI 192.168.204.128@tcp [ 6306.100388] Key type .llcrypt unregistered [ 6306.103442] Key type ._llcrypt unregistered [ 6330.302421] Key type ._llcrypt registered [ 6330.307449] Key type .llcrypt registered [ 6330.442872] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6331.320075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6331.414913] alg: No test for adler32 (adler32-zlib) [ 6332.457708] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 6332.678316] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 6334.360452] Key type lgssc registered [ 6335.413773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6343.445367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6343.461910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6345.105445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6350.111989] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6355.425912] LustreError: 197541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.451168] LustreError: 197541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6359.372337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6359.690532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6365.215398] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6373.677548] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6378.936544] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6384.822564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6385.144090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6390.259321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6390.852473] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6398.877858] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6402.973995] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6406.247421] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6420.490803] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6426.094043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6426.102063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.119185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6430.184797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6430.186634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6430.188300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6430.242784] Lustre: Skipped 1 previous similar message [ 6431.756422] Lustre: server umount lustre-OST0000 complete [ 6435.298506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.302326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6435.313932] Lustre: Skipped 1 previous similar message [ 6435.331777] Lustre: Skipped 1 previous similar message [ 6440.418545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6440.431118] Lustre: Skipped 1 previous similar message [ 6441.441170] Lustre: server umount lustre-MDT0000 complete [ 6444.844179] LustreError: 197522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627278 with bad export cookie 10965041222444393499 [ 6444.850429] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6444.856758] LustreError: 197522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6445.162446] Lustre: server umount lustre-MDT0001 complete [ 6455.023178] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 6457.683656] Key type lgssc unregistered [ 6458.008614] LNet: 201384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6458.015962] LNetError: 201384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6458.030420] LNet: Removed LNI 192.168.204.128@tcp [ 6458.791285] Key type .llcrypt unregistered [ 6458.793843] Key type ._llcrypt unregistered [ 6474.521216] Key type ._llcrypt registered [ 6474.523720] Key type .llcrypt registered [ 6474.642790] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6475.422291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6475.482097] alg: No test for adler32 (adler32-zlib) [ 6476.519708] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 6476.711955] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 6478.367175] Key type lgssc registered [ 6479.389929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6486.333614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6486.347623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6487.711617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6491.068922] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6498.277382] LustreError: 202625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6498.293493] LustreError: 202625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6498.347129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6498.693284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6503.281278] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6511.061538] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6516.323898] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6522.751237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6523.156412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6528.154885] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6533.422976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6534.826611] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6541.839501] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6551.058774] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6557.723410] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6565.928247] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6579.685590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.685907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6579.692102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6579.702894] Lustre: Skipped 1 previous similar message [ 6583.068436] Lustre: server umount lustre-OST0000 complete [ 6589.923135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.926536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6589.941352] Lustre: Skipped 1 previous similar message [ 6589.952711] Lustre: Skipped 2 previous similar messages [ 6593.290270] Lustre: server umount lustre-MDT0000 complete [ 6595.043765] LustreError: 204051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6595.080805] LustreError: 204051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6597.147037] LustreError: 205303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627430 with bad export cookie 14992347239004679165 [ 6597.151477] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6597.157131] LustreError: 205303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6597.533371] Lustre: server umount lustre-MDT0001 complete [ 6610.528908] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 6613.472960] Key type lgssc unregistered [ 6613.787081] LNet: 207594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6613.799201] LNetError: 207594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6613.837703] LNet: Removed LNI 192.168.204.128@tcp [ 6614.783311] Key type .llcrypt unregistered [ 6614.786705] Key type ._llcrypt unregistered [ 6638.009984] Key type ._llcrypt registered [ 6638.011855] Key type .llcrypt registered [ 6638.112520] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6638.870052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6638.895799] alg: No test for adler32 (adler32-zlib) [ 6639.964054] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 6640.303276] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 6642.008098] Key type lgssc registered [ 6643.302842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6651.399487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6651.418307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.868617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6657.595798] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6663.139753] LustreError: 209005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6663.157148] LustreError: 209005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6665.362144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6665.759841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6669.822145] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6677.097809] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6683.169749] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6690.190804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6690.445838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6696.019688] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6697.653772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6702.302775] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6717.410181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6717.419920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6717.439759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6718.435428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6718.445930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6718.447063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6718.491880] Lustre: Skipped 1 previous similar message [ 6720.614955] Lustre: server umount lustre-OST0000 complete [ 6727.137684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6727.142579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6727.148401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6729.960037] Lustre: server umount lustre-MDT0000 complete [ 6732.812983] LustreError: 208986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627566 with bad export cookie 3008857489450774410 [ 6732.814594] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6732.823107] LustreError: 208986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6733.043485] Lustre: server umount lustre-MDT0001 complete [ 6741.857833] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 6744.651710] Key type lgssc unregistered [ 6744.983681] LNet: 212687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6744.994130] LNetError: 212687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6745.005472] LNet: Removed LNI 192.168.204.128@tcp [ 6745.960884] Key type .llcrypt unregistered [ 6745.963988] Key type ._llcrypt unregistered [ 6763.441144] Key type ._llcrypt registered [ 6763.446150] Key type .llcrypt registered [ 6763.542773] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6764.243863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6764.339438] alg: No test for adler32 (adler32-zlib) [ 6765.438724] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 6765.775553] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 6767.471140] Key type lgssc registered [ 6768.520236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6775.373994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6775.395668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6776.802287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6780.732600] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6787.046617] LustreError: 213942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6787.056436] LustreError: 213942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6787.702711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6787.976201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6791.512964] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6797.004423] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6800.915036] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6806.363680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6806.630132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6811.970667] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6817.318705] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6821.995440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6832.608466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6832.613844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6832.618719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6832.627885] Lustre: Skipped 1 previous similar message [ 6832.638358] Lustre: Skipped 1 previous similar message [ 6837.318359] Lustre: server umount lustre-OST0000 complete [ 6842.850494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.852092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6842.859672] Lustre: Skipped 1 previous similar message [ 6846.608961] Lustre: server umount lustre-MDT0000 complete [ 6847.969780] LustreError: 213942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.986871] LustreError: 213942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6849.998319] LustreError: 213924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627683 with bad export cookie 16697858430292393859 [ 6850.006599] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6850.009760] LustreError: 213924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6850.284896] Lustre: server umount lustre-MDT0001 complete [ 6859.560583] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 6861.916751] Key type lgssc unregistered [ 6862.130520] LNet: 217632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6862.137380] LNetError: 217632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6862.150969] LNet: Removed LNI 192.168.204.128@tcp [ 6862.744386] Key type .llcrypt unregistered [ 6862.747815] Key type ._llcrypt unregistered [ 6876.302749] Key type ._llcrypt registered [ 6876.305334] Key type .llcrypt registered [ 6876.435356] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6877.416380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6877.531862] alg: No test for adler32 (adler32-zlib) [ 6878.582318] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 6878.845596] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 6880.495160] Key type lgssc registered [ 6881.594757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6888.947448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6888.962369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6890.271922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.726437] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6899.682373] LustreError: 218873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6899.707019] LustreError: 218873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6901.348312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6901.651448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6905.467605] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6912.459449] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6917.523339] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6923.417280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6923.694573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6929.101589] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6934.968206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6935.379478] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6942.659981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6942.741550] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6942.929910] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6948.356833] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6955.223860] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6956.750522] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6968.491412] Lustre: Permanently deactivating lustre-OST0001 [ 6968.509804] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6974.541384] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6985.714961] Lustre: Permanently reactivating lustre-OST0001 [ 6985.732571] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6985.737380] Lustre: Skipped 2 previous similar messages [ 6994.996965] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.011400] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6995.019380] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6995.044790] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6995.085064] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6997.726255] Lustre: server umount lustre-OST0001 complete [ 7000.044394] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7000.046529] LustreError: 222078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7000.051992] Lustre: Skipped 2 previous similar messages [ 7000.086348] LustreError: 222078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7003.619263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7003.626620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7003.642184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7005.162609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7005.164106] LustreError: 220694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7005.172486] Lustre: Skipped 1 previous similar message [ 7005.196272] LustreError: 220694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7008.083748] Lustre: server umount lustre-OST0000 complete [ 7014.372609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7014.380088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7014.393113] Lustre: Skipped 1 previous similar message [ 7014.398105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7017.981325] Lustre: server umount lustre-MDT0000 complete [ 7020.517802] LustreError: 218872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7020.535461] LustreError: 218872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7021.205862] LustreError: 218854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627854 with bad export cookie 6634470824682605759 [ 7021.216595] LustreError: 218854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7021.217242] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7021.740818] Lustre: server umount lustre-MDT0001 complete [ 7033.495331] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7036.327369] Key type lgssc unregistered [ 7036.597113] LNet: 223941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7036.604083] LNetError: 223941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7037.673360] LNet: Removed LNI 192.168.204.128@tcp [ 7038.613421] Key type .llcrypt unregistered [ 7038.616997] Key type ._llcrypt unregistered [ 7079.323412] Key type ._llcrypt registered [ 7079.325719] Key type .llcrypt registered [ 7079.429807] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7080.590988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7080.644895] alg: No test for adler32 (adler32-zlib) [ 7081.645951] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 7081.828445] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7083.511195] Key type lgssc registered [ 7084.495791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7091.577548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7091.593107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7092.835094] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7092.850541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7092.986863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7097.713846] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7106.112211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7106.210233] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7106.274328] Lustre: 226155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7106.603791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7111.407862] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7120.214603] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7126.539801] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7133.566281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7133.777890] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7134.019746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7141.622744] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7143.006609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7149.168630] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7153.131895] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7159.583710] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7167.565936] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7174.340315] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7184.670814] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7204.280314] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7204.289277] Lustre: Skipped 2 previous similar messages [ 7240.931494] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7240.937827] Lustre: Skipped 4 previous similar messages [ 7305.332732] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7305.338117] Lustre: Skipped 7 previous similar messages [ 7327.200788] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7327.205016] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7327.232981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7327.274359] Lustre: Skipped 1 previous similar message [ 7330.522656] Lustre: server umount lustre-OST0000 complete [ 7336.946522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7336.958958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7336.977420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7336.987563] Lustre: Skipped 1 previous similar message [ 7341.728631] Lustre: server umount lustre-MDT0000 complete [ 7346.802452] LustreError: 226145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628180 with bad export cookie 16042305604060522022 [ 7346.805064] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7346.811540] LustreError: 226145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7347.334782] Lustre: server umount lustre-MDT0001 complete [ 7360.265833] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7363.441685] Key type lgssc unregistered [ 7363.876944] LNet: 230966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7363.888502] LNetError: 230966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7363.907927] LNet: Removed LNI 192.168.204.128@tcp [ 7365.462676] Key type .llcrypt unregistered [ 7365.474134] Key type ._llcrypt unregistered [ 7390.220540] Key type ._llcrypt registered [ 7390.223314] Key type .llcrypt registered [ 7390.325258] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7391.581059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7391.612604] alg: No test for adler32 (adler32-zlib) [ 7392.696181] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 7392.925261] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7394.623951] Key type lgssc registered [ 7395.985396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7403.991993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7404.012466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.498254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7409.363507] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7415.783181] LustreError: 232371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7415.799555] LustreError: 232371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7417.441108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7417.812897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7421.593888] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7428.420306] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7433.349200] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7439.545685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7439.897533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7446.074355] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7451.200630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7453.502968] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7468.513455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7468.525407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7468.539923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7471.585392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7471.591641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7471.596738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7471.613770] Lustre: Skipped 1 previous similar message [ 7472.644592] Lustre: server umount lustre-OST0000 complete [ 7476.706976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7476.729026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7481.825966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7481.837069] Lustre: Skipped 2 previous similar messages [ 7482.493208] Lustre: server umount lustre-MDT0000 complete [ 7486.677288] LustreError: 232354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628320 with bad export cookie 18328774963569543578 [ 7486.682368] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7486.692486] LustreError: 232354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7487.086989] Lustre: server umount lustre-MDT0001 complete [ 7498.615852] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7501.415881] Key type lgssc unregistered [ 7501.689523] LNet: 236059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7501.699541] LNetError: 236059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7501.734967] LNet: Removed LNI 192.168.204.128@tcp [ 7502.582513] Key type .llcrypt unregistered [ 7502.585625] Key type ._llcrypt unregistered [ 7524.858589] Key type ._llcrypt registered [ 7524.860903] Key type .llcrypt registered [ 7524.983950] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7526.337846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7526.376467] alg: No test for adler32 (adler32-zlib) [ 7527.508156] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 7527.839754] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7529.647917] Key type lgssc registered [ 7530.767189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7801.785941] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7804.303590] Key type lgssc unregistered [ 7804.533670] LNet: 238075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7804.541850] LNetError: 238075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7804.559211] LNet: Removed LNI 192.168.204.128@tcp [ 7805.338617] Key type .llcrypt unregistered [ 7805.342674] Key type ._llcrypt unregistered [ 7833.424808] Key type ._llcrypt registered [ 7833.430404] Key type .llcrypt registered [ 7833.542873] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 7851.078844] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7851.998570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7852.019786] alg: No test for adler32 (adler32-zlib) [ 7853.251898] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 7853.580112] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7855.330271] Key type lgssc registered [ 7856.604493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7885.987716] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7576 sec ========== 13:18:37 (1788628717) [ 7888.029952] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:18:38 (1788628718) === [ 7891.626366] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:18:42 (1788628722) === [ 7938.904481] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7942.014188] Key type lgssc unregistered [ 7942.316451] LNet: 243313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7942.333330] LNetError: 243313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7942.353709] LNet: Removed LNI 192.168.204.128@tcp [ 7943.449617] Key type .llcrypt unregistered [ 7943.451747] Key type ._llcrypt unregistered