[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 533699409 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002262] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006565] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009124] LSM: Security Framework initializing [ 0.010053] Yama: becoming mindful. [ 0.011034] SELinux: Initializing. [ 0.012057] *** VALIDATE selinux *** [ 0.019773] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023580] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026130] *** VALIDATE tmpfs *** [ 0.027487] *** VALIDATE proc *** [ 0.029208] *** VALIDATE cgroup *** [ 0.030009] *** VALIDATE cgroup2 *** [ 0.032266] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035027] Spectre V2 : User space: Vulnerable [ 0.037005] Speculative Store Bypass: Vulnerable [ 0.039620] debug: unmapping init [mem 0xffffffff8d259000-0xffffffff8d260fff] [ 0.041175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042694] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043020] ... version: 2 [ 0.044012] ... bit width: 48 [ 0.045009] ... generic registers: 4 [ 0.045923] ... value mask: 0000ffffffffffff [ 0.046014] ... max period: 00007fffffffffff [ 0.047010] ... fixed-purpose events: 3 [ 0.048012] ... event mask: 000000070000000f [ 0.049276] rcu: Hierarchical SRCU implementation. [ 0.051317] smp: Bringing up secondary CPUs ... [ 0.052573] x86: Booting SMP configuration: [ 0.053021] .... node #0, CPUs: #1 #2 #3 [ 0.056543] smp: Brought up 1 node, 4 CPUs [ 0.057984] smpboot: Max logical packages: 1 [ 0.058014] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.128297] node 0 deferred pages initialised in 67ms [ 0.132399] devtmpfs: initialized [ 0.133175] x86/mm: Memory block size: 128MB [ 0.136507] gcov: version magic: 0x41383552 [ 0.139254] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145396] pinctrl core: initialized pinctrl subsystem [ 0.148181] [ 0.148891] ************************************************************* [ 0.152014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155011] ** ** [ 0.157011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160015] ** ** [ 0.163013] ** This means that this kernel is built to expose internal ** [ 0.165012] ** IOMMU data structures, which may compromise security on ** [ 0.168014] ** your system. ** [ 0.170011] ** ** [ 0.172010] ** If you see this message and you are not debugging the ** [ 0.174008] ** kernel, report this immediately to your vendor! ** [ 0.175009] ** ** [ 0.178013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181013] ************************************************************* [ 0.183778] NET: Registered protocol family 16 [ 0.186514] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.195125] cpuidle: using governor menu [ 0.196688] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201039] PCI: Using configuration type 1 for base access [ 0.204135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214216] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.217027] cryptd: max_cpu_qlen set to 1000 [ 0.219333] ACPI: Added _OSI(Module Device) [ 0.221014] ACPI: Added _OSI(Processor Device) [ 0.223017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.229995] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235487] ACPI: Interpreter enabled [ 0.237056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238010] ACPI: Using IOAPIC for interrupt routing [ 0.240123] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243382] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254090] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268266] acpiphp: Slot [2] registered [ 0.270113] acpiphp: Slot [5] registered [ 0.271131] acpiphp: Slot [6] registered [ 0.273121] acpiphp: Slot [7] registered [ 0.275124] acpiphp: Slot [8] registered [ 0.276094] acpiphp: Slot [9] registered [ 0.278127] acpiphp: Slot [10] registered [ 0.279118] acpiphp: Slot [3] registered [ 0.281094] acpiphp: Slot [4] registered [ 0.282083] acpiphp: Slot [11] registered [ 0.284091] acpiphp: Slot [12] registered [ 0.285097] acpiphp: Slot [13] registered [ 0.287105] acpiphp: Slot [14] registered [ 0.288067] acpiphp: Slot [15] registered [ 0.289069] acpiphp: Slot [16] registered [ 0.290094] acpiphp: Slot [17] registered [ 0.292088] acpiphp: Slot [18] registered [ 0.293093] acpiphp: Slot [19] registered [ 0.295101] acpiphp: Slot [20] registered [ 0.296130] acpiphp: Slot [21] registered [ 0.298084] acpiphp: Slot [22] registered [ 0.298994] acpiphp: Slot [23] registered [ 0.300123] acpiphp: Slot [24] registered [ 0.302102] acpiphp: Slot [25] registered [ 0.304121] acpiphp: Slot [26] registered [ 0.305114] acpiphp: Slot [27] registered [ 0.307109] acpiphp: Slot [28] registered [ 0.309166] acpiphp: Slot [29] registered [ 0.310095] acpiphp: Slot [30] registered [ 0.312114] acpiphp: Slot [31] registered [ 0.313071] PCI host bridge to bus 0000:00 [ 0.315017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.317024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.319019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.322023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.325024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.328024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.330168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334215] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338349] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.352939] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.358017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364631] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.368090] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.371049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.374025] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.379015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.391016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.397010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402974] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.410990] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.416014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.431014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.441970] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.453016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.471016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.480876] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.487015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.494013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.515014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.525811] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.533014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.543016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.562017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.573573] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.582012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.589014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.605015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.615000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.624015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.636016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.651016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.660875] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.663307] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.665297] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.668338] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.671233] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.676216] iommu: Default domain type: Passthrough [ 0.679393] SCSI subsystem initialized [ 0.681150] ACPI: bus type USB registered [ 0.684104] usbcore: registered new interface driver usbfs [ 0.686108] usbcore: registered new interface driver hub [ 0.688086] usbcore: registered new device driver usb [ 0.690173] pps_core: LinuxPPS API ver. 1 registered [ 0.693012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.696058] PTP clock support registered [ 0.699054] EDAC MC: Ver: 3.0.0 [ 0.701194] PCI: Using ACPI for IRQ routing [ 0.702959] NetLabel: Initializing [ 0.703011] NetLabel: domain hash size = 128 [ 0.704010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.705099] NetLabel: unlabeled traffic allowed by default [ 0.707215] vgaarb: loaded [ 0.709272] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.710011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.716328] clocksource: Switched to clocksource kvm-clock [ 0.819245] VFS: Disk quotas dquot_6.6.0 [ 0.820882] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.823967] *** VALIDATE ramfs *** [ 0.825135] *** VALIDATE hugetlbfs *** [ 0.826447] pnp: PnP ACPI init [ 0.829036] pnp: PnP ACPI: found 6 devices [ 0.857394] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.861209] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.863519] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.866063] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.868765] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.871467] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.874299] NET: Registered protocol family 2 [ 0.877222] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.882946] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.888211] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.895060] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.899588] TCP: Hash tables configured (established 65536 bind 65536) [ 0.903174] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.907201] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.910580] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.913505] NET: Registered protocol family 1 [ 0.916103] RPC: Registered named UNIX socket transport module. [ 0.918892] RPC: Registered udp transport module. [ 0.920899] RPC: Registered tcp transport module. [ 0.923128] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.925816] NET: Registered protocol family 44 [ 0.927646] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.930394] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.933127] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.935677] PCI: CLS 0 bytes, default 64 [ 0.937480] Unpacking initramfs... [ 2.350357] debug: unmapping init [mem 0xffff8ec7bcc54000-0xffff8ec7bffbffff] [ 2.354821] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.357912] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.361362] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.855869] Initialise system trusted keyrings [ 2.859070] Key type blacklist registered [ 2.860876] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.874367] zbud: loaded [ 2.878982] *** VALIDATE nfs *** [ 2.880139] *** VALIDATE nfs4 *** [ 2.881834] pstore: using deflate compression [ 2.885118] Platform Keyring initialized [ 3.009560] NET: Registered protocol family 38 [ 3.011658] Key type asymmetric registered [ 3.013524] Asymmetric key parser 'x509' registered [ 3.015513] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.019170] io scheduler mq-deadline registered [ 3.020762] io scheduler kyber registered [ 3.022392] io scheduler bfq registered [ 3.024500] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.027760] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.031263] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.034096] ACPI: Power Button [PWRF] [ 3.039754] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.048408] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.069640] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.078415] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.097649] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.125403] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.153905] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.159058] Non-volatile memory driver v1.3 [ 3.160387] Linux agpgart interface v0.103 [ 3.190146] virtio_blk virtio1: [vda] 150648 512-byte logical blocks (77.1 MB/73.6 MiB) [ 3.193477] vda: detected capacity change from 0 to 77131776 [ 3.210629] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.214214] vdb: detected capacity change from 0 to 1073741824 [ 3.234139] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.237603] vdc: detected capacity change from 0 to 2621440000 [ 3.254515] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.258513] vdd: detected capacity change from 0 to 2621440000 [ 3.274368] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.278052] vde: detected capacity change from 0 to 4294967296 [ 3.306930] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.310240] vdf: detected capacity change from 0 to 4294967296 [ 3.319336] libphy: Fixed MDIO Bus: probed [ 3.326703] usbcore: registered new interface driver usbserial_generic [ 3.328894] usbserial: USB Serial support registered for generic [ 3.331860] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.336620] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.337996] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.340138] mousedev: PS/2 mouse device common for all mice [ 3.345215] rtc_cmos 00:05: RTC can wake from S4 [ 3.347765] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.349513] rtc_cmos 00:05: registered as rtc0 [ 3.356626] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.358862] intel_pstate: CPU model not supported [ 3.359643] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.363927] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.364063] hid: raw HID events driver (C) Jiri Kosina [ 3.367802] usbcore: registered new interface driver usbhid [ 3.369439] usbhid: USB HID core driver [ 3.371057] drop_monitor: Initializing network drop monitor service [ 3.373635] Initializing XFRM netlink socket [ 3.375028] NET: Registered protocol family 10 [ 3.378813] Segment Routing with IPv6 [ 3.380104] NET: Registered protocol family 17 [ 3.381919] mpls_gso: MPLS GSO support [ 3.386138] RAS: Correctable Errors collector initialized. [ 3.387751] AVX version of gcm_enc/dec engaged. [ 3.389841] AES CTR mode by8 optimization enabled [ 3.473779] sched_clock: Marking stable (3473755779, 0)->(4364236101, -890480322) [ 3.478691] registered taskstats version 1 [ 3.481201] Loading compiled-in X.509 certificates [ 3.483815] zswap: loaded using pool lzo/zbud [ 3.507865] Key type big_key registered [ 3.523100] Key type encrypted registered [ 3.524955] ima: No TPM chip found, activating TPM-bypass! [ 3.527609] ima: Allocated hash algorithm: sha1 [ 3.529461] ima: No architecture policies found [ 3.531192] evm: Initialising EVM extended attributes: [ 3.532637] evm: security.selinux [ 3.533834] evm: security.ima [ 3.534730] evm: security.capability [ 3.536154] evm: HMAC attrs: 0x1 [ 3.538727] rtc_cmos 00:05: setting system clock to 2026-09-30 02:12:41 UTC (1790734361) [ 3.545218] debug: unmapping init [mem 0xffffffff8e203000-0xffffffff8e3fffff] [ 3.547885] debug: unmapping init [mem 0xffffffff8cf82000-0xffffffff8d258fff] [ 3.556142] Write protecting the kernel read-only data: 28672k [ 3.559133] debug: unmapping init [mem 0xffffffff8b603000-0xffffffff8b7fffff] [ 3.562104] debug: unmapping init [mem 0xffffffff8bf14000-0xffffffff8bffffff] [ 3.600730] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.609873] systemd[1]: Detected virtualization kvm. [ 3.611866] systemd[1]: Detected architecture x86-64. [ 3.614059] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.643389] systemd[1]: No hostname configured. [ 3.645442] systemd[1]: Set hostname to . [ 3.648080] random: systemd: uninitialized urandom read (16 bytes read) [ 3.650474] systemd[1]: Initializing machine ID from random generator. [ 3.709996] random: ln: uninitialized urandom read (6 bytes read) [ 3.798604] random: systemd: uninitialized urandom read (16 bytes read) [ 3.801379] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.806044] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.811162] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.467930] device-mapper: uevent: version 1.0.3 [ 4.470830] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.197167] virtio_net virtio0 ens2: renamed from eth0 [ 5.201268] random: fast init done [ 5.211989] scsi host0: ata_piix [ 5.253116] scsi host1: ata_piix [ 5.255320] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.258368] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.046293] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.042214] random: crng init done [ 10.043731] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.547368] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.746528] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.011938] SELinux: Disabled at runtime. [ 12.071963] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.081534] systemd[1]: Detected virtualization kvm. [ 12.084645] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.614587] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.619783] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.627729] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.632895] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.636369] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.644157] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.656412] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ 12.792865] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.163130] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.590395] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.752217] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.835918] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.867376] EDAC sbridge: Ver: 1.1.2 [ 15.469553] Key type dns_resolver registered [ 15.808493] NFS: Registering the id_resolver key type [ 15.811073] Key type id_resolver registered [ 15.813072] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg342-server login: [ 60.199648] hrtimer: interrupt took 5940003 ns [ 69.649562] libcfs: loading out-of-tree module taints kernel. [ 69.694295] Key type ._llcrypt registered [ 69.696871] Key type .llcrypt registered [ 69.861728] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 89.372426] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 91.068436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.095425] alg: No test for adler32 (adler32-zlib) [ 92.773683] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 93.833540] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 95.624227] Key type lgssc registered [ 97.628911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 117.997931] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 156.690519] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 169.179568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 169.222599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.448963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 170.516950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 170.585562] Lustre: lustre-MDT0000: new disk, initializing [ 170.660402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.688824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 175.598602] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.915514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.067261] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 192.127227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 192.131917] Lustre: Skipped 1 previous similar message [ 192.216825] Lustre: lustre-MDT0001: new disk, initializing [ 192.295901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.334072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 192.345478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.296596] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.296118] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 212.154100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.373512] Lustre: lustre-OST0000: new disk, initializing [ 212.376850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 212.389069] Lustre: 8445:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.459975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.834622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 215.841107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 215.930121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 217.981510] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.864335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.170719] Lustre: lustre-OST0001: new disk, initializing [ 235.179669] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 235.197512] Lustre: 9519:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.331921] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 242.301702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 242.321121] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 242.496107] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.682310] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.223689] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 261.812055] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 268.295707] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing check_logdir /tmp/testlogs/ [ 273.281706] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing yml_node [ 277.462611] Lustre: DEBUG MARKER: Client: 2.17.59.3 [ 279.932572] Lustre: DEBUG MARKER: MDS: 2.17.59.3 [ 282.168195] Lustre: DEBUG MARKER: OSS: 2.17.59.3 [ 283.679415] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 29 22:17:20 EDT 2026 [ 298.948878] Lustre: DEBUG MARKER: only running tests: 0-31 [ 300.638578] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 302.074535] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 319.472636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.472947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.484771] Lustre: Skipped 2 previous similar messages [ 319.505997] Lustre: Skipped 3 previous similar messages [ 323.830619] Lustre: server umount lustre-MDT0000 complete [ 327.493920] LustreError: 9518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790734685 with bad export cookie 8754645393396753427 [ 327.494840] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 327.505518] LustreError: 9518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 329.697639] LustreError: 9527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.705942] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.710788] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.720786] LustreError: 9527:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 329.753467] Lustre: Skipped 2 previous similar messages [ 332.715943] LustreError: 6514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.716858] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.732911] LustreError: 6514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 332.739931] Lustre: Skipped 2 previous similar messages [ 334.165926] Lustre: server umount lustre-MDT0001 complete [ 338.259219] Lustre: server umount lustre-OST0000 complete [ 342.146429] Lustre: server umount lustre-OST0001 complete [ 348.619606] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 358.641178] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 392.126223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 392.373393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 392.412263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 392.499976] Lustre: lustre-MDT0000: new disk, initializing [ 392.590103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 392.617369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 397.210819] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 407.496068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.642823] Lustre: 15123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 407.695224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 407.703266] Lustre: Skipped 1 previous similar message [ 407.751652] Lustre: lustre-MDT0001: new disk, initializing [ 407.830238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 407.871165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 407.894311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 411.890975] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 421.201937] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 426.676345] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 433.300294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 433.532742] Lustre: lustre-OST0000: new disk, initializing [ 433.536777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 433.543399] Lustre: 17090:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 433.639268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.975920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 434.981715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 435.119122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 439.987895] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 449.972464] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 455.302992] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 455.501514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 460.051497] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 460.212566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 465.889806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.903558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.928692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.935428] Lustre: Skipped 1 previous similar message [ 468.419078] Lustre: server umount lustre-OST0000 complete [ 474.595741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 474.602313] LustreError: Skipped 1 previous similar message [ 474.605773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.615457] Lustre: Skipped 1 previous similar message [ 474.626285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 485.351940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 485.359965] Lustre: Skipped 5 previous similar messages [ 486.368423] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 486.726950] Lustre: server umount lustre-MDT0000 complete [ 489.818803] LustreError: 15115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790734847 with bad export cookie 8754645393396754960 [ 489.823834] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.834903] LustreError: 15115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 490.111049] Lustre: server umount lustre-MDT0001 complete [ 492.847609] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:20:50 (1790734850) [ 499.521912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.019265] LustreError: 19356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.120814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.304790] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 505.313270] LustreError: 19357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.421852] LustreError: 19356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.301732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 514.536952] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 519.690518] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 526.737499] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 533.039748] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 539.566379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 546.067632] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 553.235712] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 569.829460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 569.832025] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.846160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 569.889100] Lustre: Skipped 2 previous similar messages [ 574.775559] Lustre: server umount lustre-OST0000 complete [ 580.065685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.077645] Lustre: Skipped 1 previous similar message [ 592.352126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 592.456091] Lustre: server umount lustre-MDT0000 complete [ 594.402640] LustreError: 19357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.429277] LustreError: 19357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 595.363363] LustreError: 22154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790734953 with bad export cookie 8754645393396755835 [ 595.365205] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.373223] LustreError: 22154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 595.837168] Lustre: server umount lustre-MDT0001 complete [ 606.587587] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 608.900189] Key type lgssc unregistered [ 609.080602] LNet: 22907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.090722] LNetError: 22907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.107597] LNet: Removed LNI 192.168.203.142@tcp [ 609.818169] Key type .llcrypt unregistered [ 609.822177] Key type ._llcrypt unregistered [ 630.610409] Key type ._llcrypt registered [ 630.615268] Key type .llcrypt registered [ 630.751891] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 631.920797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 632.128400] alg: No test for adler32 (adler32-zlib) [ 633.236532] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 633.508708] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 635.184200] Key type lgssc registered [ 636.351674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 643.571846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 643.590593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.057758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.300311] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 654.305553] LustreError: 24334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.329279] LustreError: 24334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 657.631134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.897980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 662.293473] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 669.283480] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 674.313707] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 680.981470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 681.294339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 686.674170] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 689.537553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 693.303733] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 713.185651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 713.198201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.215514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 715.232981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 715.237236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 715.242214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.535905] Lustre: server umount lustre-OST0000 complete [ 720.353664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.359808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 720.383506] Lustre: Skipped 1 previous similar message [ 720.387776] Lustre: Skipped 1 previous similar message [ 725.476955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.492951] Lustre: Skipped 2 previous similar messages [ 725.845183] Lustre: server umount lustre-MDT0000 complete [ 729.219902] LustreError: 24315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790735087 with bad export cookie 18376555858630649621 [ 729.223865] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 729.233048] LustreError: 24315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 729.449965] Lustre: server umount lustre-MDT0001 complete [ 741.421465] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 745.100588] Key type lgssc unregistered [ 745.506729] LNet: 28307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.515907] LNetError: 28307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.535207] LNet: Removed LNI 192.168.203.142@tcp [ 747.450268] Key type .llcrypt unregistered [ 747.452888] Key type ._llcrypt unregistered [ 769.467572] Key type ._llcrypt registered [ 769.474188] Key type .llcrypt registered [ 769.553948] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 770.487710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 770.594403] alg: No test for adler32 (adler32-zlib) [ 771.793756] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 772.079031] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 773.771041] Key type lgssc registered [ 774.900180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 782.601967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 782.652364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.313029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 788.553272] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 794.596454] LustreError: 29716:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 794.634852] LustreError: 29716:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 796.435029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 796.763980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 800.924912] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 807.455759] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 812.977315] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 826.447177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 826.890837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 830.020253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 833.249950] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 840.683835] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 850.402564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 850.414598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.438249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 851.429248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 851.432112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 851.444911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.852537] Lustre: server umount lustre-OST0000 complete [ 856.560743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.592643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 856.595663] Lustre: Skipped 1 previous similar message [ 861.667258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 861.670994] Lustre: Skipped 1 previous similar message [ 862.655509] Lustre: server umount lustre-MDT0000 complete [ 865.951446] LustreError: 29697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790735223 with bad export cookie 7346618182913220775 [ 865.954595] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.968202] LustreError: 29697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 866.313876] Lustre: server umount lustre-MDT0001 complete [ 878.140258] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 880.536891] Key type lgssc unregistered [ 880.944789] LNet: 33593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.956854] LNetError: 33593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.978863] LNet: Removed LNI 192.168.203.142@tcp [ 881.902399] Key type .llcrypt unregistered [ 881.906284] Key type ._llcrypt unregistered [ 904.558743] Key type ._llcrypt registered [ 904.561501] Key type .llcrypt registered [ 904.681906] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 905.623770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 905.736308] alg: No test for adler32 (adler32-zlib) [ 906.944399] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 907.266804] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 908.937585] Key type lgssc registered [ 909.957328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 916.482868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 916.495802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.883429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 921.829221] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 927.211400] LustreError: 35015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.229746] LustreError: 35015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 930.132360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.420360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 934.619927] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 941.159770] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 947.176057] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 953.535770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 953.870445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.420568] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 962.112213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 968.244735] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 981.474078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 981.482054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.500687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 982.504914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 982.506531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 982.535542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 984.204864] Lustre: server umount lustre-OST0000 complete [ 991.720996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 991.725302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.741161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 991.744745] Lustre: Skipped 1 previous similar message [ 994.415751] Lustre: server umount lustre-MDT0000 complete [ 997.868430] LustreError: 35016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.895346] LustreError: 35016:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 999.086489] LustreError: 37596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790735357 with bad export cookie 6292052828979350799 [ 999.106684] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 999.113755] LustreError: 37596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 999.674933] Lustre: server umount lustre-MDT0001 complete [ 1011.775691] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1014.220868] Key type lgssc unregistered [ 1014.480887] LNet: 38548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.508872] LNetError: 38548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.534690] LNet: Removed LNI 192.168.203.142@tcp [ 1015.348624] Key type .llcrypt unregistered [ 1015.350996] Key type ._llcrypt unregistered [ 1036.549987] Key type ._llcrypt registered [ 1036.553464] Key type .llcrypt registered [ 1036.639460] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1037.589805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1037.639492] alg: No test for adler32 (adler32-zlib) [ 1038.721192] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 1038.967984] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1040.593123] Key type lgssc registered [ 1041.801541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1049.784045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1049.822386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.395622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1056.532658] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1061.865126] LustreError: 39976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.884339] LustreError: 39976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1065.342692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.767635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1069.942414] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1076.776078] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1081.989249] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1089.585781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1089.956797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1096.120986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1096.634225] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1104.488543] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1111.388409] Lustre: lustre-OST0000: Not available for connect from 192.168.203.42@tcp (stopping) [ 1111.526033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1111.526753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.537411] LustreError: Skipped 1 previous similar message [ 1111.557849] Lustre: Skipped 1 previous similar message [ 1116.408060] Lustre: server umount lustre-OST0000 complete [ 1137.125953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.130260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.146257] Lustre: Skipped 1 previous similar message [ 1137.169684] Lustre: Skipped 3 previous similar messages [ 1142.242736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.250973] Lustre: Skipped 1 previous similar message [ 1142.851831] Lustre: server umount lustre-MDT0000 complete [ 1147.209285] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790735505 with bad export cookie 12257277849704166459 [ 1147.212286] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1147.226592] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1153.973393] Lustre: server umount lustre-MDT0001 complete [ 1166.696770] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1169.989931] Key type lgssc unregistered [ 1170.346602] LNet: 43685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.358997] LNetError: 43685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.387365] LNet: Removed LNI 192.168.203.142@tcp [ 1171.341589] Key type .llcrypt unregistered [ 1171.343510] Key type ._llcrypt unregistered [ 1195.016744] Key type ._llcrypt registered [ 1195.051882] Key type .llcrypt registered [ 1195.139121] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1196.377342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1196.396994] alg: No test for adler32 (adler32-zlib) [ 1197.564419] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 1197.905458] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1199.680408] Key type lgssc registered [ 1201.092930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1209.593869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1209.611557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.921246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.473892] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1220.076082] LustreError: 45112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1220.103584] LustreError: 45112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1224.196909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.481481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1229.165855] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1236.387873] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1241.205400] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1247.269729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1247.716030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1253.077731] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1255.921599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1260.144694] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1265.633935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1265.642237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1265.655757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.162308] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.170258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.174392] Lustre: Skipped 1 previous similar message [ 1266.183085] Lustre: Skipped 1 previous similar message [ 1267.844437] Lustre: server umount lustre-MDT0000 complete [ 1271.271705] LustreError: 45113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.292798] LustreError: 45113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1271.445895] LustreError: 45095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790735629 with bad export cookie 5706336311745513779 [ 1271.452896] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1271.462439] LustreError: 45095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1272.144107] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 1272.161917] Lustre: Skipped 2 previous similar messages [ 1276.386257] LustreError: 45109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.386597] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1276.415580] LustreError: 45109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1276.438369] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1278.072568] Lustre: server umount lustre-MDT0001 complete [ 1307.104231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1307.375506] Lustre: server umount lustre-OST0000 complete [ 1325.511063] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1328.481809] Key type lgssc unregistered [ 1328.778557] LNet: 48975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1328.794694] LNetError: 48975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.811560] LNet: Removed LNI 192.168.203.142@tcp [ 1329.877281] Key type .llcrypt unregistered [ 1329.883696] Key type ._llcrypt unregistered [ 1355.675767] Key type ._llcrypt registered [ 1355.681584] Key type .llcrypt registered [ 1355.836599] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1357.269896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1357.310148] alg: No test for adler32 (adler32-zlib) [ 1358.381626] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 1358.691523] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1360.368211] Key type lgssc registered [ 1361.625991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1369.737196] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1369.766619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1385.376388] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1390.497693] LustreError: 50378:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 1417.248200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.115540] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1433.605698] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1597.954768] Lustre: server umount lustre-OST0000 complete [ 1613.179572] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1616.041484] Key type lgssc unregistered [ 1616.392528] LNet: 52052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.403528] LNetError: 52052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.419920] LNet: Removed LNI 192.168.203.142@tcp [ 1617.297465] Key type .llcrypt unregistered [ 1617.302810] Key type ._llcrypt unregistered [ 1641.937837] Key type ._llcrypt registered [ 1641.940320] Key type .llcrypt registered [ 1642.080480] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1643.063608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1643.188116] alg: No test for adler32 (adler32-zlib) [ 1644.345933] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 1644.627845] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1646.352257] Key type lgssc registered [ 1647.481976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1654.707152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1654.742060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1656.191664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.621986] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1666.532610] LustreError: 53485:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1666.564638] LustreError: 53485:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1668.322645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1668.680385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1673.095583] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1680.006475] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1685.468459] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1691.728230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1692.195855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.576986] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1701.428427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1704.439409] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1708.513919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1708.525803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1708.538736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1711.589374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1711.597886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1711.606063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1716.712754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1716.722929] Lustre: Skipped 1 previous similar message [ 1721.312581] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1721.481448] Lustre: server umount lustre-OST0000 complete [ 1726.945944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.952114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1726.966368] Lustre: Skipped 1 previous similar message [ 1726.988688] Lustre: Skipped 2 previous similar messages [ 1731.573903] Lustre: server umount lustre-MDT0000 complete [ 1732.070470] LustreError: 53479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1732.102344] LustreError: 53479:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1734.707279] LustreError: 53465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736092 with bad export cookie 7570350980092177281 [ 1734.712106] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1734.724534] LustreError: 53465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1735.073266] Lustre: server umount lustre-MDT0001 complete [ 1744.750845] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1747.774159] Key type lgssc unregistered [ 1748.145829] LNet: 57012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.153039] LNetError: 57012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.222364] LNet: Removed LNI 192.168.203.142@tcp [ 1750.637938] Key type .llcrypt unregistered [ 1750.643901] Key type ._llcrypt unregistered [ 1775.427869] Key type ._llcrypt registered [ 1775.431209] Key type .llcrypt registered [ 1775.542493] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1776.741191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1776.778237] alg: No test for adler32 (adler32-zlib) [ 1777.884447] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 1778.133673] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1779.817112] Key type lgssc registered [ 1781.032865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1789.816172] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1789.861059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1805.472380] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1810.592461] LustreError: 58413:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 1837.344415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.285424] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1851.070202] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1858.242508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.061626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1859.168932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1863.941489] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1864.166160] LustreError: 58439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1864.192205] LustreError: 58439:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1869.291057] LustreError: 58440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1872.364903] LustreError: 59451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1872.389259] LustreError: 59451:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1872.926353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1873.476386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1878.500255] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1886.158936] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1891.583796] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1893.875612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1893.878831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.898326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1893.940698] Lustre: Skipped 1 previous similar message [ 1898.996771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1899.009604] Lustre: Skipped 2 previous similar messages [ 1899.838584] Lustre: server umount lustre-OST0000 complete [ 1924.579450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.580234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1924.582295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1924.591708] Lustre: Skipped 1 previous similar message [ 1928.025953] Lustre: server umount lustre-MDT0000 complete [ 1930.728066] LustreError: 59455:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1930.750413] LustreError: 59455:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1931.758471] LustreError: 58420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736289 with bad export cookie 8802720583522381648 [ 1931.780166] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1931.783460] LustreError: 58420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1938.332078] Lustre: server umount lustre-MDT0001 complete [ 1950.115929] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1953.254357] Key type lgssc unregistered [ 1953.636377] LNet: 62156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.647745] LNetError: 62156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1953.672155] LNet: Removed LNI 192.168.203.142@tcp [ 1954.605232] Key type .llcrypt unregistered [ 1954.608476] Key type ._llcrypt unregistered [ 1979.803888] Key type ._llcrypt registered [ 1979.808842] Key type .llcrypt registered [ 1979.914747] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1980.844090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.958956] alg: No test for adler32 (adler32-zlib) [ 1982.050369] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 1982.330040] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1983.976198] Key type lgssc registered [ 1985.156746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1993.825685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1993.858131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.370413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2000.175572] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2005.987310] LustreError: 63586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2006.018903] LustreError: 63586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2008.735138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.125746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2014.629767] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2022.731888] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2028.796663] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2036.051295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2036.413690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.499560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2042.919457] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2050.793610] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2070.499403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.518561] LustreError: 65409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2070.519989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2070.531771] Lustre: Skipped 1 previous similar message [ 2070.596840] LustreError: 65409:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2070.710749] Lustre: server umount lustre-OST0000 complete [ 2075.622530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2075.622994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2075.629480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2075.635208] Lustre: Skipped 1 previous similar message [ 2080.737548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2080.743087] Lustre: Skipped 2 previous similar messages [ 2081.031702] Lustre: server umount lustre-MDT0000 complete [ 2084.702696] LustreError: 63568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736442 with bad export cookie 14491516619244072081 [ 2084.709326] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2084.723194] LustreError: 63568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2085.139982] Lustre: server umount lustre-MDT0001 complete [ 2096.031925] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2098.781389] Key type lgssc unregistered [ 2099.042741] LNet: 67140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2099.049701] LNetError: 67140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2099.076608] LNet: Removed LNI 192.168.203.142@tcp [ 2100.025109] Key type .llcrypt unregistered [ 2100.036707] Key type ._llcrypt unregistered [ 2123.200457] Key type ._llcrypt registered [ 2123.203171] Key type .llcrypt registered [ 2123.324817] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2124.225817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2124.274019] alg: No test for adler32 (adler32-zlib) [ 2125.383956] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 2125.681904] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2127.392908] Key type lgssc registered [ 2128.440914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2135.976641] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:48:13 (1790736493) [ 2142.155991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2142.177462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.663604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2146.740797] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2153.872774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.956532] LustreError: 68838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.978457] LustreError: 68838:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2154.215230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.947113] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2166.740147] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2173.509847] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2179.432812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2179.803822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.291803] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2189.070872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2191.792073] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2201.196674] Lustre: Failing over lustre-MDT0000 [ 2201.620867] Lustre: server umount lustre-MDT0000 complete [ 2204.524369] LustreError: 68834:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.646974] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2204.662956] Lustre: Skipped 1 previous similar message [ 2209.598439] LustreError: 68834:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.638124] LustreError: 68834:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2210.527453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.670554] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2210.887658] Lustre: *** cfs_fail_loc=135, val=0*** [ 2210.993477] LustreError: 71925:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2210.996822] LustreError: 71925:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -2 [ 2211.002749] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2211.002749] [ 2211.011777] LustreError: MGC192.168.203.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2211.020889] LustreError: 71913:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2211.027468] LustreError: 71913:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2211.032721] LustreError: 71913:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2211.039899] LustreError: 71913:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2211.062935] Lustre: server umount lustre-MDT0000 complete [ 2211.068481] LustreError: 71913:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2214.711314] LustreError: 68835:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.731619] LustreError: 68835:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2217.759691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.953956] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2217.971164] LustreError: 68831:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2218.305342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2218.357990] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2219.832147] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2223.260996] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2223.592602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2223.622398] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2223.666647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2233.840684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.843308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2233.854911] Lustre: Skipped 2 previous similar messages [ 2236.727655] Lustre: server umount lustre-OST0000 complete [ 2244.066419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.067755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2244.068903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2244.068909] Lustre: Skipped 1 previous similar message [ 2244.074620] Lustre: Skipped 1 previous similar message [ 2249.188876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.199152] Lustre: Skipped 2 previous similar messages [ 2253.282839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2253.292497] Lustre: Skipped 1 previous similar message [ 2254.304130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2254.507945] Lustre: server umount lustre-MDT0000 complete [ 2258.411362] LustreError: 69234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.434791] LustreError: 69234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2258.626086] LustreError: 68819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736616 with bad export cookie 4702509109650913135 [ 2258.643422] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.644679] LustreError: 68819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2259.375663] Lustre: server umount lustre-MDT0001 complete [ 2270.911687] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2273.891687] Key type lgssc unregistered [ 2274.163078] LNet: 73704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.175856] LNetError: 73704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.215206] LNet: Removed LNI 192.168.203.142@tcp [ 2275.306411] Key type .llcrypt unregistered [ 2275.309898] Key type ._llcrypt unregistered [ 2299.562960] Key type ._llcrypt registered [ 2299.567449] Key type .llcrypt registered [ 2299.723428] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2301.092862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2301.146280] alg: No test for adler32 (adler32-zlib) [ 2302.688040] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 2303.093199] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2304.831363] Key type lgssc registered [ 2306.408823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2314.373361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2314.399578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.878628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.840931] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2325.991400] LustreError: 75139:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.021970] LustreError: 75139:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2328.685904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2329.079151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2333.943878] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2341.415599] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2346.906822] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2353.734544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2354.250598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.373052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2360.473674] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2367.102486] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2376.977585] Lustre: Failing over lustre-MDT0000 [ 2377.250894] Lustre: server umount lustre-MDT0000 complete [ 2378.725819] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.748795] LustreError: 75139:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.753460] Lustre: Skipped 1 previous similar message [ 2378.784228] LustreError: 75139:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2380.139848] LustreError: 75133:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.844435] LustreError: 75135:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.859216] LustreError: 75135:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2385.671722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.828369] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2385.920665] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2386.031969] LustreError: 78225:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2386.045983] LustreError: 78225:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -9 [ 2386.067337] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2386.067337] [ 2386.085227] LustreError: MGC192.168.203.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2386.098448] LustreError: 78214:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2386.117390] LustreError: 78214:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2386.129310] LustreError: 78214:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2386.139031] LustreError: 78214:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2386.161127] Lustre: server umount lustre-MDT0000 complete [ 2386.165869] LustreError: 78214:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2388.962795] LustreError: 76139:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.977473] LustreError: 76139:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2392.183681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.360742] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2392.630215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.701810] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2395.450282] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2396.670215] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2397.694642] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2397.723031] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2397.803113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2407.907983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2407.912847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2407.930025] Lustre: Skipped 2 previous similar messages [ 2411.855679] Lustre: server umount lustre-OST0000 complete [ 2418.156860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2418.165424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2418.187243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2418.194721] Lustre: Skipped 1 previous similar message [ 2418.203407] Lustre: Skipped 2 previous similar messages [ 2422.721779] Lustre: server umount lustre-MDT0000 complete [ 2423.266162] LustreError: 75139:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.290321] LustreError: 75139:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2426.270618] LustreError: 75874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736784 with bad export cookie 8343374717666130691 [ 2426.279585] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2426.280155] LustreError: 75874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2426.552246] Lustre: server umount lustre-MDT0001 complete [ 2437.236717] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2439.618429] Key type lgssc unregistered [ 2439.874574] LNet: 80001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.881587] LNetError: 80001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2439.931911] LNet: Removed LNI 192.168.203.142@tcp [ 2440.731830] Key type .llcrypt unregistered [ 2440.734388] Key type ._llcrypt unregistered [ 2453.083955] Key type ._llcrypt registered [ 2453.086921] Key type .llcrypt registered [ 2453.193929] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2453.531526] Key type .llcrypt unregistered [ 2453.534862] Key type ._llcrypt unregistered [ 2475.275591] Key type ._llcrypt registered [ 2475.278229] Key type .llcrypt registered [ 2475.368557] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2476.351283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2476.478695] alg: No test for adler32 (adler32-zlib) [ 2477.548562] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 2477.816052] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2479.496239] Key type lgssc registered [ 2480.528630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2487.010305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2487.033573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.559263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.383421] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2499.046204] LustreError: 81920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.086773] LustreError: 81920:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2502.859821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.414264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2507.623862] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2514.888832] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2521.420383] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2527.903942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2528.232842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.114725] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2534.407619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2542.785454] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2553.826444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2553.838308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.853964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2554.850477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.880299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2554.881929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2554.893744] Lustre: Skipped 1 previous similar message [ 2558.510711] Lustre: server umount lustre-OST0000 complete [ 2565.104626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.109833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2565.110285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2565.121037] Lustre: Skipped 1 previous similar message [ 2565.134243] Lustre: Skipped 1 previous similar message [ 2568.832987] Lustre: server umount lustre-MDT0000 complete [ 2570.211981] LustreError: 83353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.243622] LustreError: 83353:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2572.325240] LustreError: 81901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736930 with bad export cookie 10254602602074597361 [ 2572.333639] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2572.339472] LustreError: 81901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2572.598459] Lustre: server umount lustre-MDT0001 complete [ 2583.209428] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2585.738715] Key type lgssc unregistered [ 2585.980668] LNet: 85453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2585.989748] LNetError: 85453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2586.007895] LNet: Removed LNI 192.168.203.142@tcp [ 2586.802138] Key type .llcrypt unregistered [ 2586.812098] Key type ._llcrypt unregistered [ 2608.594817] Key type ._llcrypt registered [ 2608.604576] Key type .llcrypt registered [ 2608.740589] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2609.786906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2609.820223] alg: No test for adler32 (adler32-zlib) [ 2610.988720] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 2611.272570] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2612.984165] Key type lgssc registered [ 2614.041499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2621.985421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2622.056327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.626511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.082522] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2633.256086] LustreError: 86879:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.277620] LustreError: 86879:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2635.919326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2636.187584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2640.558246] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2647.184815] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2652.761304] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2660.133697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2660.657706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.379919] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2668.852379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2675.365599] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2682.338587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2682.348684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.365865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2684.387727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2684.389277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2687.532896] Lustre: server umount lustre-OST0000 complete [ 2692.579637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2692.593193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.618316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2692.630670] Lustre: Skipped 1 previous similar message [ 2694.625258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2694.625421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.752108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.763384] Lustre: Skipped 2 previous similar messages [ 2705.378397] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2705.540674] Lustre: server umount lustre-MDT0000 complete [ 2708.993209] LustreError: 86860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790737066 with bad export cookie 5697704355774467497 [ 2709.006017] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2709.006758] LustreError: 86860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2709.553411] Lustre: server umount lustre-MDT0001 complete [ 2721.346531] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2723.955490] Key type lgssc unregistered [ 2724.279928] LNet: 90413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2724.292681] LNetError: 90413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2724.318658] LNet: Removed LNI 192.168.203.142@tcp [ 2725.156948] Key type .llcrypt unregistered [ 2725.162446] Key type ._llcrypt unregistered [ 2748.142623] Key type ._llcrypt registered [ 2748.145846] Key type .llcrypt registered [ 2748.242471] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2749.614782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2749.715497] alg: No test for adler32 (adler32-zlib) [ 2750.773397] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 2751.029741] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2752.792967] Key type lgssc registered [ 2753.808816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2761.290712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2761.321662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.663696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.662961] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2772.967049] LustreError: 91840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.002491] LustreError: 91840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2776.234127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.634216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2782.142949] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2789.787913] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2794.776528] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2801.994982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2802.435136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2807.562312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2808.657919] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2816.012382] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2831.844190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2831.852718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.866916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2832.291308] Lustre: server umount lustre-OST0000 complete [ 2837.474509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.479685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2837.493713] Lustre: Skipped 1 previous similar message [ 2837.503044] Lustre: Skipped 1 previous similar message [ 2842.598148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2842.606139] Lustre: Skipped 1 previous similar message [ 2847.717218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2847.725967] Lustre: Skipped 1 previous similar message [ 2848.740286] Lustre: 91144:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790737190/real 1790737190] req@0000000053d4931b x1877723953786240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790737206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2848.797864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2850.272280] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2850.463942] Lustre: server umount lustre-MDT0000 complete [ 2853.856637] Lustre: 91143:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790737195/real 1790737195] req@00000000da57c28a x1877723953788032/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790737211 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2853.860718] LustreError: 91841:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.913479] Lustre: 91143:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2853.965914] LustreError: 91841:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2855.255262] LustreError: 91823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790737213 with bad export cookie 11053885920446429554 [ 2855.266666] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2855.281458] LustreError: 91823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2855.801536] Lustre: server umount lustre-MDT0001 complete [ 2868.493504] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2871.828680] Key type lgssc unregistered [ 2872.161121] LNet: 95396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2872.170271] LNetError: 95396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2872.195202] LNet: Removed LNI 192.168.203.142@tcp [ 2873.098490] Key type .llcrypt unregistered [ 2873.108994] Key type ._llcrypt unregistered [ 2896.466949] Key type ._llcrypt registered [ 2896.468802] Key type .llcrypt registered [ 2896.595969] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2897.680084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2897.721682] alg: No test for adler32 (adler32-zlib) [ 2898.788182] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 2899.030240] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2900.712302] Key type lgssc registered [ 2901.953263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2909.127308] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2909.145484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2924.704450] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2929.824958] LustreError: 96826:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 2956.513607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2962.177245] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2969.504595] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2975.072404] Lustre: server umount lustre-OST0000 complete [ 2983.267948] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 23:02:20 (1790737340) [ 2989.479711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2989.874830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.481975] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2995.171623] LustreError: 98465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.194782] LustreError: 98465:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2999.274036] LustreError: 98466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.689120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3003.079768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3007.340445] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3014.552622] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3019.872950] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3025.688368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3026.069714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.468372] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3034.316942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3038.475553] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3049.956533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3049.964318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3049.986825] Lustre: Skipped 1 previous similar message [ 3049.987838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3050.001689] Lustre: Skipped 1 previous similar message [ 3055.074375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3055.082830] Lustre: Skipped 1 previous similar message [ 3055.587742] Lustre: server umount lustre-OST0000 complete [ 3059.691677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3059.704042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.722407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3065.273210] Lustre: server umount lustre-MDT0000 complete [ 3065.313655] LustreError: 98466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.336984] LustreError: 98466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3068.980384] LustreError: 98445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790737426 with bad export cookie 9438258829941397069 [ 3068.981434] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3068.993265] LustreError: 98445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3069.056685] LustreError: 96140:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 3069.483132] Lustre: server umount lustre-MDT0001 complete [ 3080.334982] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3082.720994] Key type lgssc unregistered [ 3082.923890] LNet: 102153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3082.930087] LNetError: 102153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3082.947764] LNet: Removed LNI 192.168.203.142@tcp [ 3083.585212] Key type .llcrypt unregistered [ 3083.587796] Key type ._llcrypt unregistered [ 3123.582457] Key type ._llcrypt registered [ 3123.584914] Key type .llcrypt registered [ 3123.693721] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3124.937760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3125.002447] alg: No test for adler32 (adler32-zlib) [ 3126.107580] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 3126.369418] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3128.064563] Key type lgssc registered [ 3129.249735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3136.807663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3136.849624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.319386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3142.044066] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3148.779085] LustreError: 104357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.798911] LustreError: 104357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3148.949340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.272963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3153.470062] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3160.914847] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3167.147245] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3174.206814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3174.627253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3182.168173] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3183.866136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3190.168245] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3201.514766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3201.520524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3201.543193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3204.584913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3204.591034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3204.606590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.063348] Lustre: server umount lustre-OST0000 complete [ 3210.727189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3210.738176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.759566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3210.763586] Lustre: Skipped 1 previous similar message [ 3214.818724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3214.820340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.076368] Lustre: server umount lustre-MDT0000 complete [ 3219.812106] LustreError: 104336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790737577 with bad export cookie 7766205017865833180 [ 3219.823300] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3220.200856] Lustre: server umount lustre-MDT0001 complete [ 3231.169470] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3234.156379] Key type lgssc unregistered [ 3234.413687] LNet: 107888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3234.426416] LNetError: 107888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3234.448452] LNet: Removed LNI 192.168.203.142@tcp [ 3235.289425] Key type .llcrypt unregistered [ 3235.291453] Key type ._llcrypt unregistered [ 3253.043131] Key type ._llcrypt registered [ 3253.045383] Key type .llcrypt registered [ 3253.192890] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3254.388589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3254.504234] alg: No test for adler32 (adler32-zlib) [ 3255.586214] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 3255.896409] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3257.624555] Key type lgssc registered [ 3258.858884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3266.444225] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3266.477039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3282.016635] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3287.136413] LustreError: 109146:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 3313.760540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3319.858573] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3327.341932] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3334.188848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.562664] LustreError: MGC192.168.203.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3334.593653] LustreError: 110170:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3334.606904] LustreError: 110170:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3334.623812] LustreError: 110170:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3334.733730] Lustre: server umount lustre-MDT0000 complete [ 3334.742064] LustreError: 110170:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3353.211568] Lustre: server umount lustre-OST0000 complete [ 3362.171596] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3372.037318] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3405.209668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3405.418800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3405.451601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3405.530541] Lustre: lustre-MDT0000: new disk, initializing [ 3405.600942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.617433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3409.199442] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3418.666494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3418.807855] Lustre: 112964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3418.873654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3418.878645] Lustre: Skipped 1 previous similar message [ 3419.018905] Lustre: lustre-MDT0001: new disk, initializing [ 3419.132694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3419.196504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3419.214332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3424.568781] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3434.282771] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3440.396510] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3447.549911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3447.859121] Lustre: lustre-OST0000: new disk, initializing [ 3447.862551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3447.869677] Lustre: 114932:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3447.975053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3454.205630] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3457.564854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3457.573518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3457.758287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3464.247821] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3469.000424] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3469.162399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3473.571191] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3473.768125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3477.986184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3477.999803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3478.017505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3478.503983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3478.519750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3478.552088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3478.557565] Lustre: Skipped 1 previous similar message [ 3481.599437] Lustre: server umount lustre-OST0000 complete [ 3485.669128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3485.675613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.696587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3488.742925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3488.750623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3488.778241] Lustre: Skipped 1 previous similar message [ 3491.776791] Lustre: server umount lustre-MDT0000 complete [ 3494.818185] LustreError: 112955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790737852 with bad export cookie 6265125904615174415 [ 3494.823815] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3494.827773] LustreError: 112955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3495.098522] Lustre: server umount lustre-MDT0001 complete [ 3503.312601] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 23:11:00 (1790737860) [ 3506.268185] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3521.656577] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3530.700664] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3562.271278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.485537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3562.515832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3562.587433] Lustre: lustre-MDT0000: new disk, initializing [ 3562.664878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3562.682724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3566.823320] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3576.292141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.380727] Lustre: 119830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3576.427457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3576.431412] Lustre: Skipped 1 previous similar message [ 3576.506627] Lustre: lustre-MDT0001: new disk, initializing [ 3576.586185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3576.639438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3576.654337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3580.935735] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3590.281544] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3595.900107] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3602.840875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3603.056698] Lustre: lustre-OST0000: new disk, initializing [ 3603.070842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3603.076773] Lustre: 121797:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3603.185021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3609.684475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3609.699825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3609.729489] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3609.802117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3618.846877] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3623.336851] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3623.550210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3628.137665] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3628.330108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3630.564673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3630.574288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3630.577631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3630.585597] Lustre: Skipped 1 previous similar message [ 3630.599041] Lustre: Skipped 1 previous similar message [ 3640.819959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3640.829338] Lustre: Skipped 3 previous similar messages [ 3644.385939] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3644.567465] Lustre: server umount lustre-OST0000 complete [ 3648.482967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3648.489284] LustreError: Skipped 1 previous similar message [ 3648.497102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.450835] Lustre: server umount lustre-MDT0000 complete [ 3656.162164] LustreError: 120953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3656.188656] LustreError: 120953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3657.938032] LustreError: 120623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738015 with bad export cookie 6265125904615175297 [ 3657.940833] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3657.950373] LustreError: 120623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3658.326477] Lustre: server umount lustre-MDT0001 complete [ 3665.984072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.497759] LustreError: 124014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.638228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3671.425624] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3672.045297] LustreError: 124015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.133775] LustreError: 124014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3679.256518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.397971] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3689.856641] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3694.522681] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3700.599800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3706.998833] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3713.397605] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3720.096667] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3725.794163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3725.802294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3725.816558] Lustre: Skipped 1 previous similar message [ 3725.820443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3725.824131] Lustre: Skipped 3 previous similar messages [ 3728.059385] Lustre: server umount lustre-OST0000 complete [ 3738.904394] Lustre: server umount lustre-MDT0000 complete [ 3742.182401] LustreError: 126505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3742.208929] LustreError: 126505:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3742.608797] LustreError: 123995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738100 with bad export cookie 6265125904615176179 [ 3742.626219] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3742.937747] Lustre: server umount lustre-MDT0001 complete [ 3752.575700] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3755.345934] Key type lgssc unregistered [ 3755.625954] LNet: 127644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3755.633716] LNetError: 127644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3755.645186] LNet: Removed LNI 192.168.203.142@tcp [ 3756.432638] Key type .llcrypt unregistered [ 3756.439409] Key type ._llcrypt unregistered [ 3773.281518] Key type ._llcrypt registered [ 3773.283583] Key type .llcrypt registered [ 3773.407773] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3787.623563] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3788.624705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3788.653921] alg: No test for adler32 (adler32-zlib) [ 3789.646961] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 3789.947043] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3791.664197] Key type lgssc registered [ 3792.688697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3830.164525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3830.190525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3831.422256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3831.449891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3831.553428] Lustre: lustre-MDT0000: new disk, initializing [ 3831.627639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3831.651682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3835.829813] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3844.332850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3844.434235] Lustre: 130812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3844.477677] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3844.480526] Lustre: Skipped 1 previous similar message [ 3844.535773] Lustre: lustre-MDT0001: new disk, initializing [ 3844.608624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3844.636476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3844.646605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3848.161474] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3855.740713] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3859.997107] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3865.384351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3865.593417] Lustre: lustre-OST0000: new disk, initializing [ 3865.596899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3865.603565] Lustre: 132780:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3865.678980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3871.292163] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3874.405000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3874.416171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3874.571278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3879.849169] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3884.393865] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3884.606615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3889.093568] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3889.309745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3895.266226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3895.269549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.280549] LustreError: Skipped 1 previous similar message [ 3895.298839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3895.308362] Lustre: Skipped 1 previous similar message [ 3897.348814] Lustre: server umount lustre-OST0000 complete [ 3900.897259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3900.905351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.920618] Lustre: Skipped 1 previous similar message [ 3900.933468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3905.513158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3905.513836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.625097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3910.627266] Lustre: Skipped 1 previous similar message [ 3915.232415] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3915.466409] Lustre: server umount lustre-MDT0000 complete [ 3919.057731] LustreError: 130805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738277 with bad export cookie 18277597327582927817 [ 3919.063940] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3919.071184] LustreError: 130805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3919.386100] Lustre: server umount lustre-MDT0001 complete [ 3928.678956] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 23:18:05 (1790738285) [ 3934.154617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.683492] LustreError: 135192:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3934.710976] LustreError: 135192:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3934.757523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3938.480634] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3939.811407] LustreError: 135193:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3944.933592] LustreError: 135192:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3945.957572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3946.170819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3949.973498] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3955.987047] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3961.670557] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3964.899401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.920718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3964.937708] Lustre: Skipped 1 previous similar message [ 3964.965397] Lustre: Skipped 2 previous similar messages [ 3969.944944] Lustre: server umount lustre-MDT0000 complete [ 3970.022714] LustreError: 135192:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3970.077631] LustreError: 135192:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3975.077294] LustreError: 135173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738333 with bad export cookie 18277597327582928699 [ 3975.098984] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3975.589663] Lustre: server umount lustre-MDT0001 complete [ 3984.752302] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 23:19:01 (1790738341) [ 3990.967670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4006.624700] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4011.744511] LustreError: 137606:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 4037.473723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4044.610611] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4052.884297] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4055.795760] Lustre: server umount lustre-OST0000 complete [ 4064.101727] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 23:20:21 (1790738421) [ 4070.391227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.880744] LustreError: 139054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.899357] LustreError: 139054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4070.953936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4075.708503] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4080.105024] LustreError: 139055:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4080.125223] LustreError: 139055:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4083.871721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4088.984804] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4096.266417] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4101.587673] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4107.515567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4107.828338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4107.832387] Lustre: Skipped 1 previous similar message [ 4115.249996] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4123.899896] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4135.393294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4135.402465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.416317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.934184] Lustre: server umount lustre-MDT0000 complete [ 4141.538273] LustreError: 139051:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.562807] LustreError: 139051:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4141.770680] LustreError: 139035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738499 with bad export cookie 18277597327582929364 [ 4141.778432] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4141.788907] LustreError: 139035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4142.242581] Lustre: server umount lustre-MDT0001 complete [ 4156.938485] Lustre: server umount lustre-OST0000 complete [ 4164.697493] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 23:22:01 (1790738521) [ 4171.125675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4172.265382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4177.385210] LustreError: 142705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4177.415916] LustreError: 142705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4178.200438] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4188.174331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4193.847315] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4201.700341] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4207.461815] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4214.597247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4217.974068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4221.259347] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4229.188878] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4233.801062] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4233.997546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4237.999959] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4238.228222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4243.942327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4243.944552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.946022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4243.946029] Lustre: Skipped 3 previous similar messages [ 4243.960132] LustreError: Skipped 1 previous similar message [ 4244.014011] Lustre: Skipped 3 previous similar messages [ 4246.460963] Lustre: server umount lustre-OST0000 complete [ 4256.429584] LustreError: 146016:0:(ldlm_resource.c:1227:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (0000000017aacdaa) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4258.529215] Lustre: server umount lustre-MDT0000 complete [ 4259.303324] LustreError: 142705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.329259] LustreError: 142705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4262.321379] LustreError: 142686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738620 with bad export cookie 18277597327582930435 [ 4262.340336] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4262.345453] LustreError: 142686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4262.649237] Lustre: server umount lustre-MDT0001 complete [ 4270.957268] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 23:23:47 (1790738627) [ 4277.770668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4293.344629] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4298.464394] LustreError: 146863:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 4324.192973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4324.206894] Lustre: Skipped 2 previous similar messages [ 4330.354451] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4337.632752] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4343.915122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.331984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4349.306528] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4357.455615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.420863] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4369.439462] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4375.674849] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4380.539131] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4380.791487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4385.773577] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4386.006709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4388.327870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4388.331500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.363562] Lustre: Skipped 2 previous similar messages [ 4388.374372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4388.377100] Lustre: Skipped 3 previous similar messages [ 4394.311927] Lustre: server umount lustre-MDT0000 complete [ 4394.995764] LustreError: 146889:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.035403] LustreError: 146889:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4398.048705] LustreError: 146870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738756 with bad export cookie 18277597327582931457 [ 4398.049549] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4398.062461] LustreError: 146870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4398.550740] Lustre: server umount lustre-MDT0001 complete [ 4412.954644] Lustre: server umount lustre-OST0000 complete [ 4420.782987] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 23:26:17 (1790738777) [ 4427.408571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4443.040525] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4448.165261] LustreError: 151067:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 4479.758771] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4486.635618] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4493.768390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.315920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4498.924387] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4507.786719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4513.692921] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4522.145442] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4528.518801] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4535.856959] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4536.001567] Lustre: lustre-OST0001: new disk, initializing [ 4536.007037] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4536.020713] Lustre: 153939:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4541.546659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4541.569097] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4541.748595] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4543.695445] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4553.673250] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4558.944707] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4559.222221] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4564.950195] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4565.129949] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4567.533114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4567.550458] Lustre: Skipped 2 previous similar messages [ 4567.554858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4567.565766] Lustre: Skipped 4 previous similar messages [ 4581.345055] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4581.440068] Lustre: server umount lustre-OST0000 complete [ 4588.007076] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4588.025468] LustreError: Skipped 1 previous similar message [ 4599.264108] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4599.600808] Lustre: server umount lustre-OST0001 complete [ 4605.410060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4610.570980] Lustre: server umount lustre-MDT0000 complete [ 4613.401148] LustreError: 151075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738971 with bad export cookie 18277597327582932409 [ 4613.405695] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4613.419322] LustreError: 151075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4613.727726] Lustre: server umount lustre-MDT0001 complete [ 4635.554064] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 23:29:52 (1790738992) [ 4636.941741] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4638.531034] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4640.146617] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 23:29:57 (1790738997) [ 4647.069254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.449179] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4647.503223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4647.778178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.797981] Lustre: Skipped 6 previous similar messages [ 4652.873082] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4661.662940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4661.848631] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4661.897333] Lustre: 157498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4666.593371] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4673.635862] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4679.182344] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4685.601122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4685.776309] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4691.756332] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4694.195056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4698.967423] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4704.029862] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4704.249115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4708.593809] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4708.814893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4714.465401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4714.485550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4714.518307] Lustre: Skipped 5 previous similar messages [ 4714.532472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4714.542080] Lustre: Skipped 13 previous similar messages [ 4716.789296] Lustre: server umount lustre-OST0000 complete [ 4737.304385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4743.596213] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4750.711550] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4755.565545] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4756.008225] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4756.022578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4756.030482] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4756.051688] Lustre: Skipped 1 previous similar message [ 4756.916962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4761.116501] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4761.397937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4766.509545] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa0bf10eee800.ost_server_uuid 50 [ 4768.062816] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa0bf10eee800.ost_server_uuid in FULL state after 0 sec [ 4768.324088] LustreError: 158304:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.42@tcp) returned error from blocking AST (req@00000000ca95555d x1877725043493760 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000e3267712/0xfda711ee3a35b713 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.42@tcp remote: 0xee75e42ba6b36796 expref: 5 pid: 158304 timeout: 4868 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4768.369492] LustreError: lustre-MDT0000: A client on nid 192.168.203.42@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4768.377966] LustreError: 157496:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.42@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000e3267712/0xfda711ee3a35b713 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.42@tcp remote: 0xee75e42ba6b36796 expref: 6 pid: 158304 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4780.818976] Lustre: server umount lustre-OST0000 complete [ 4790.242518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4791.636180] Lustre: server umount lustre-MDT0000 complete [ 4795.379912] LustreError: 158244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739153 with bad export cookie 18277597327582933585 [ 4795.384132] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4795.392231] LustreError: 158244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4801.889524] Lustre: server umount lustre-MDT0001 complete [ 4812.234605] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4815.110632] Key type lgssc unregistered [ 4815.429322] LNet: 163623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4815.438619] LNetError: 163623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4815.460243] LNet: Removed LNI 192.168.203.142@tcp [ 4816.284218] Key type .llcrypt unregistered [ 4816.288147] Key type ._llcrypt unregistered [ 4836.979816] Key type ._llcrypt registered [ 4836.982709] Key type .llcrypt registered [ 4837.088697] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4837.952052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4837.973223] alg: No test for adler32 (adler32-zlib) [ 4839.107458] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 4839.414042] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4841.120530] Key type lgssc registered [ 4842.435093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4850.292937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4850.332664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4851.854566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4856.010970] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4857.315285] LustreError: 165060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4857.332756] LustreError: 165060:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4861.424245] LustreError: 165059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4864.063515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.510446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4868.432293] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4875.711673] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4880.888419] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4887.742910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4888.082622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4896.007277] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4896.345426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4904.657950] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4909.887578] Lustre: Failing over lustre-MDT0000 [ 4910.178202] Lustre: server umount lustre-MDT0000 complete [ 4910.561434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4910.571982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4910.609933] LustreError: 165059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.588634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.705678] LustreError: 166103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4916.721635] LustreError: 166103:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4921.840708] LustreError: 165059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.860676] LustreError: 165059:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4922.381715] LustreError: 166903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.42@tcp arrived at 1790739280 with bad export cookie 17896407839545485656 [ 4922.397795] LustreError: 166903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4927.968476] Lustre: 164362:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790739269/real 1790739269] req@00000000d7ef15cb x1877726144261248/t0(0) o400->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790739285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4927.996440] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4929.472267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4933.091872] LustreError: 166899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4933.111267] LustreError: 166899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4937.220416] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf85cd01c5f9fe9ba [ 4937.227972] Lustre: MGC192.168.203.142@tcp: Connection restored to 0@lo (at 0@lo) [ 4937.524082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4937.631831] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4941.398362] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4942.823396] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4942.845370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4943.319568] Lustre: lustre-MDT0000: Denying connection for new client 55cc32c9-d6f7-4b61-8dfd-14655adfd9e3 (at 192.168.203.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 4953.061950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.085677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4953.097989] Lustre: Skipped 2 previous similar messages [ 4953.132571] Lustre: Skipped 1 previous similar message [ 4957.574206] Lustre: server umount lustre-OST0000 complete [ 4961.417255] LustreError: 168791:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4961.436106] Lustre: 168139:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4961.452716] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4961.486888] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4961.510814] LustreError: 168139:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009cc091d2 x1877726144280576/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4961.528651] LustreError: 168139:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4961.536342] LustreError: 168139:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4961.544348] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 4961.888672] Lustre: server umount lustre-MDT0000 complete [ 4963.303806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4963.323182] LustreError: 165055:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4963.340219] LustreError: 165055:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4965.580072] LustreError: 165040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739323 with bad export cookie 17896407839545485754 [ 4965.584256] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4965.589904] LustreError: 165040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4967.995704] Lustre: server umount lustre-MDT0001 complete [ 4978.370986] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4980.871315] Key type lgssc unregistered [ 4981.070706] LNet: 169543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4981.076522] LNetError: 169543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4981.089661] LNet: Removed LNI 192.168.203.142@tcp [ 4981.711246] Key type .llcrypt unregistered [ 4981.721756] Key type ._llcrypt unregistered [ 5005.700428] Key type ._llcrypt registered [ 5005.703450] Key type .llcrypt registered [ 5005.809933] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5006.555895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5006.690313] alg: No test for adler32 (adler32-zlib) [ 5007.662603] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 5007.969475] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5009.656223] Key type lgssc registered [ 5011.030809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5019.167737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5019.187543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.522413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5024.314535] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5030.886724] LustreError: 170970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5030.901418] LustreError: 170970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5032.243698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5032.563565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5036.834610] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5043.800688] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5049.901896] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5057.198238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5057.510587] LustreError: 172792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5057.726199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.209818] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5065.212886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5072.878086] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5078.497871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5078.508793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5078.522059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5080.542871] Lustre: server umount lustre-OST0000 complete [ 5085.671112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.673799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5085.699413] Lustre: Skipped 1 previous similar message [ 5090.785975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5090.796660] Lustre: Skipped 2 previous similar messages [ 5094.972646] Lustre: server umount lustre-MDT0000 complete [ 5095.909026] LustreError: 170971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5095.935282] LustreError: 170971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5099.122958] LustreError: 170952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739457 with bad export cookie 17275372555699616900 [ 5099.133904] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5099.417428] Lustre: server umount lustre-MDT0001 complete [ 5112.417359] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5115.226155] Key type lgssc unregistered [ 5115.546662] LNet: 174503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5115.559349] LNetError: 174503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5115.580973] LNet: Removed LNI 192.168.203.142@tcp [ 5116.396037] Key type .llcrypt unregistered [ 5116.399443] Key type ._llcrypt unregistered [ 5149.919079] Key type ._llcrypt registered [ 5149.920546] Key type .llcrypt registered [ 5150.052930] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5151.364703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5151.431475] alg: No test for adler32 (adler32-zlib) [ 5152.501441] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 5152.733040] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5154.400227] Key type lgssc registered [ 5155.472828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5163.340408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5163.358074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.798850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.741819] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5174.255758] LustreError: 176221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5174.270046] LustreError: 176221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5177.153417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.568692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5181.742391] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5189.015366] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5194.929441] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5202.120903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5202.348598] LustreError: 178042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.485169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5209.026865] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5209.584651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5218.048516] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5228.007089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5228.012390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5228.022340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5230.053666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5230.056987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5230.073867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5235.173458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5235.184870] Lustre: Skipped 1 previous similar message [ 5240.289552] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5240.297566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5240.314812] Lustre: Skipped 1 previous similar message [ 5240.599581] Lustre: server umount lustre-OST0000 complete [ 5244.386451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5244.396803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.417743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5244.428605] Lustre: Skipped 1 previous similar message [ 5250.835873] Lustre: server umount lustre-MDT0000 complete [ 5254.861485] LustreError: 176201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739612 with bad export cookie 8836008890053119276 [ 5254.870594] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5254.881718] LustreError: 176201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5255.349722] Lustre: server umount lustre-MDT0001 complete [ 5268.295189] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5271.411512] Key type lgssc unregistered [ 5271.767697] LNet: 179753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5271.781099] LNetError: 179753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5272.806846] LNet: Removed LNI 192.168.203.142@tcp [ 5273.777529] Key type .llcrypt unregistered [ 5273.781221] Key type ._llcrypt unregistered [ 5297.721547] Key type ._llcrypt registered [ 5297.729127] Key type .llcrypt registered [ 5297.842924] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5299.039767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5299.186491] alg: No test for adler32 (adler32-zlib) [ 5300.337032] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 5300.651694] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5302.368202] Key type lgssc registered [ 5303.697700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5314.327339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5314.353697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.707101] Lustre: *** cfs_fail_loc=135, val=0*** [ 5315.816511] LustreError: 181294:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5315.823950] LustreError: 181294:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -2 [ 5315.835329] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5315.835329] [ 5315.844172] LustreError: MGC192.168.203.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5315.869216] LustreError: 181256:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5315.878374] LustreError: 181256:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5315.905527] LustreError: 181256:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5315.916354] LustreError: 181256:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5316.038392] Lustre: server umount lustre-MDT0000 complete [ 5316.041561] LustreError: 181256:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5322.851681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.369508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5327.998308] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5333.990977] LustreError: 181665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5334.011058] LustreError: 181665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5336.233880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5336.709702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5341.435343] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5349.254589] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5354.801911] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5357.025919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5357.033509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5357.044962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5358.562016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5358.567806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5361.699505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5361.713742] Lustre: Skipped 2 previous similar messages [ 5362.948248] Lustre: server umount lustre-MDT0000 complete [ 5365.793501] LustreError: 181648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739723 with bad export cookie 1042848412942570859 [ 5365.800981] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5365.801837] LustreError: 181648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5366.137977] Lustre: server umount lustre-MDT0001 complete [ 5376.114862] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5378.808214] Key type lgssc unregistered [ 5379.100769] LNet: 183987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5379.110974] LNetError: 183987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5379.137079] LNet: Removed LNI 192.168.203.142@tcp [ 5380.131297] Key type .llcrypt unregistered [ 5380.139308] Key type ._llcrypt unregistered [ 5401.713429] Key type ._llcrypt registered [ 5401.718594] Key type .llcrypt registered [ 5401.868723] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5402.194383] Key type .llcrypt unregistered [ 5402.198229] Key type ._llcrypt unregistered [ 5419.592332] Key type ._llcrypt registered [ 5419.594541] Key type .llcrypt registered [ 5419.693277] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5420.972875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5421.019732] alg: No test for adler32 (adler32-zlib) [ 5422.141673] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 5422.442516] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5424.113701] Key type lgssc registered [ 5425.515734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5433.354347] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5433.379514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5449.056377] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5454.241001] LustreError: 186139:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 5481.007747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5488.095379] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5496.117814] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5501.591278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.400758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5502.520185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5507.174104] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5507.556706] LustreError: 186165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5507.584435] LustreError: 186165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5512.691296] LustreError: 186164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5515.752081] LustreError: 187178:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5515.776794] LustreError: 187178:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5516.261192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5516.713364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5521.654414] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5529.277446] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5535.312600] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5538.893712] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5548.513313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5548.519309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5548.537112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5552.612324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5552.618776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5552.630058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5554.431710] Lustre: server umount lustre-OST0000 complete [ 5562.851115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5562.863202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5562.865034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5562.865040] Lustre: Skipped 1 previous similar message [ 5562.904661] Lustre: Skipped 1 previous similar message [ 5564.430734] Lustre: server umount lustre-MDT0000 complete [ 5567.975801] LustreError: 187181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5567.990135] LustreError: 187181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5568.683567] LustreError: 188801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739926 with bad export cookie 497515020712313755 [ 5568.685959] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5568.697546] LustreError: 188801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5568.937218] Lustre: server umount lustre-MDT0001 complete [ 5579.727305] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5582.310433] Key type lgssc unregistered [ 5582.719119] LNet: 190057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5582.726116] LNetError: 190057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5582.745823] LNet: Removed LNI 192.168.203.142@tcp [ 5583.726911] Key type .llcrypt unregistered [ 5583.729168] Key type ._llcrypt unregistered [ 5606.042293] Key type ._llcrypt registered [ 5606.048190] Key type .llcrypt registered [ 5606.180148] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5607.076575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5607.299852] alg: No test for adler32 (adler32-zlib) [ 5608.350387] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 5608.599317] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5610.304213] Key type lgssc registered [ 5611.473539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5618.316220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5618.336829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5619.766273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5623.969440] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5629.931090] LustreError: 191484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.956296] LustreError: 191484:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5632.000768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.513205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5637.467811] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5645.132976] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5651.124705] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5657.966893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5658.450911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.637773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5664.832461] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5672.857375] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5678.111225] Lustre: Failing over lustre-MDT0000 [ 5678.561960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5678.570303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.601601] LustreError: 191484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5678.856140] Lustre: server umount lustre-MDT0000 complete [ 5683.171351] LustreError: 192472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5683.176361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5683.201908] LustreError: 192472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5683.232788] Lustre: Skipped 1 previous similar message [ 5688.289546] LustreError: 191483:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.316899] LustreError: 191483:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 5693.413159] LustreError: 191484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.433903] LustreError: 191484:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5697.615666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.791384] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5698.045902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5698.108439] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5699.894753] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5702.039754] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5703.151431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5703.179946] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5703.217614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5706.047411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5709.525908] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5718.499484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.511471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5718.520240] Lustre: Skipped 1 previous similar message [ 5718.544029] Lustre: Skipped 1 previous similar message [ 5723.621609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5723.630366] Lustre: Skipped 1 previous similar message [ 5724.448788] Lustre: server umount lustre-OST0000 complete [ 5728.741536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.745252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5728.769837] Lustre: Skipped 1 previous similar message [ 5733.858363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5733.866524] Lustre: Skipped 2 previous similar messages [ 5738.978228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5738.990051] Lustre: Skipped 1 previous similar message [ 5742.051546] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5742.493544] Lustre: server umount lustre-MDT0000 complete [ 5744.097524] LustreError: 191484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5744.124328] LustreError: 191484:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5746.376286] LustreError: 191464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740104 with bad export cookie 17646045638229751551 [ 5746.386357] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5746.398047] LustreError: 191464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5746.614876] Lustre: server umount lustre-MDT0001 complete [ 5756.679199] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5759.436174] Key type lgssc unregistered [ 5759.764118] LNet: 196269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5759.773318] LNetError: 196269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5760.820529] LNet: Removed LNI 192.168.203.142@tcp [ 5761.753734] Key type .llcrypt unregistered [ 5761.755688] Key type ._llcrypt unregistered [ 5787.624275] Key type ._llcrypt registered [ 5787.626516] Key type .llcrypt registered [ 5787.781555] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5788.675727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5788.692114] alg: No test for adler32 (adler32-zlib) [ 5789.700981] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 5789.920650] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5791.592224] Key type lgssc registered [ 5793.401977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5803.612631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5803.671800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5805.580421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.310724] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5815.780085] LustreError: 197699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5815.809645] LustreError: 197699:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5819.888385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5820.430574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5825.133831] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5832.730835] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5837.018465] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5842.741254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5843.042265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5848.936494] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5850.254775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5857.751829] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5861.404217] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5869.483882] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5885.965242] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5901.794854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5901.796385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5901.828207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5901.849021] Lustre: Skipped 1 previous similar message [ 5905.531288] Lustre: server umount lustre-OST0000 complete [ 5912.035135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5912.038810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5912.060941] Lustre: Skipped 2 previous similar messages [ 5915.013205] Lustre: server umount lustre-MDT0000 complete [ 5917.153201] LustreError: 198428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5917.182766] LustreError: 198428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5918.765887] LustreError: 197678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740276 with bad export cookie 9668105982022903381 [ 5918.771866] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5918.776382] LustreError: 197678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5919.069857] Lustre: server umount lustre-MDT0001 complete [ 5930.530149] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5933.071671] Key type lgssc unregistered [ 5933.350320] LNet: 201590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5933.362965] LNetError: 201590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5933.399083] LNet: Removed LNI 192.168.203.142@tcp [ 5934.390845] Key type .llcrypt unregistered [ 5934.392766] Key type ._llcrypt unregistered [ 5952.598876] Key type ._llcrypt registered [ 5952.600519] Key type .llcrypt registered [ 5952.687863] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5953.578272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5953.766625] alg: No test for adler32 (adler32-zlib) [ 5954.771098] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 5955.048785] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5956.810155] Key type lgssc registered [ 5957.915709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5965.707130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5965.744905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5967.277334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.750984] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5977.580177] LustreError: 202871:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5977.626237] LustreError: 202871:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5982.690816] LustreError: 202872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.948512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5983.432915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5989.311523] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5998.202204] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6003.700455] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6010.639688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6011.074424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6012.118379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6017.418028] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6025.103221] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6030.164832] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6045.177753] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6053.307480] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6060.886746] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6069.729665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6069.731907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.741483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6069.748157] Lustre: Skipped 1 previous similar message [ 6069.760592] Lustre: Skipped 1 previous similar message [ 6074.852717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6074.872350] Lustre: Skipped 1 previous similar message [ 6075.520029] Lustre: server umount lustre-OST0000 complete [ 6079.971475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.986494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6080.001949] Lustre: Skipped 1 previous similar message [ 6080.018481] Lustre: Skipped 1 previous similar message [ 6085.091192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6085.099797] Lustre: Skipped 1 previous similar message [ 6090.210631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6090.218972] Lustre: Skipped 1 previous similar message [ 6093.280105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6093.403576] Lustre: server umount lustre-MDT0000 complete [ 6095.330713] LustreError: 206798:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6095.361603] LustreError: 206798:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6096.209306] LustreError: 205550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740454 with bad export cookie 1574816973182833511 [ 6096.210876] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6096.220984] LustreError: 205550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6096.556177] Lustre: server umount lustre-MDT0001 complete [ 6107.916795] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6110.778418] Key type lgssc unregistered [ 6111.063544] LNet: 207891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6111.074984] LNetError: 207891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6111.089970] LNet: Removed LNI 192.168.203.142@tcp [ 6112.028969] Key type .llcrypt unregistered [ 6112.030753] Key type ._llcrypt unregistered [ 6137.919291] Key type ._llcrypt registered [ 6137.923501] Key type .llcrypt registered [ 6138.063899] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6138.961369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6138.976476] alg: No test for adler32 (adler32-zlib) [ 6140.191546] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 6140.474630] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6142.152190] Key type lgssc registered [ 6143.189307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6151.356458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6151.377623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6152.844943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6157.394273] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6162.405301] LustreError: 209329:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.428494] LustreError: 209329:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6165.094584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6165.585500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6170.146575] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6177.738669] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6184.285891] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6191.997995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6192.488271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6197.638891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6198.885386] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6207.329772] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6223.330945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6223.341721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.362484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.402365] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6226.402872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.416126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6229.490725] Lustre: server umount lustre-OST0000 complete [ 6236.642720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.646289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6236.662397] Lustre: Skipped 1 previous similar message [ 6236.674865] Lustre: Skipped 2 previous similar messages [ 6239.890043] Lustre: server umount lustre-MDT0000 complete [ 6241.766221] LustreError: 209324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6241.784855] LustreError: 209324:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6243.647967] LustreError: 209310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740601 with bad export cookie 17980985791363441072 [ 6243.655538] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6243.661050] LustreError: 209310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6244.068984] Lustre: server umount lustre-MDT0001 complete [ 6256.455477] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6259.597499] Key type lgssc unregistered [ 6259.918444] LNet: 213013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6259.935856] LNetError: 213013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6259.957868] LNet: Removed LNI 192.168.203.142@tcp [ 6260.902835] Key type .llcrypt unregistered [ 6260.909321] Key type ._llcrypt unregistered [ 6279.194727] Key type ._llcrypt registered [ 6279.197856] Key type .llcrypt registered [ 6279.286899] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6280.943235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6281.051160] alg: No test for adler32 (adler32-zlib) [ 6282.442686] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 6282.753490] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6284.544214] Key type lgssc registered [ 6286.716454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6295.648634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6295.691496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6297.323582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6302.943571] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6307.816599] LustreError: 214294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6307.827414] LustreError: 214294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6312.567215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.006899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6317.633700] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6326.503639] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6332.788075] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6339.998705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6340.411320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.469704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6346.809927] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6354.210367] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6373.348466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6373.372524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6373.381915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6373.393498] Lustre: Skipped 1 previous similar message [ 6375.753970] Lustre: server umount lustre-OST0000 complete [ 6383.588425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6383.617679] Lustre: Skipped 1 previous similar message [ 6383.630484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6383.637806] Lustre: Skipped 1 previous similar message [ 6388.706156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6388.713115] Lustre: Skipped 1 previous similar message [ 6393.828078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6393.839596] Lustre: Skipped 1 previous similar message [ 6394.337731] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6394.555671] Lustre: server umount lustre-MDT0000 complete [ 6398.468358] LustreError: 214275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740756 with bad export cookie 16340790433700472701 [ 6398.482694] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6398.785732] Lustre: server umount lustre-MDT0001 complete [ 6412.033897] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6415.300396] Key type lgssc unregistered [ 6415.660402] LNet: 217978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6415.667960] LNetError: 217978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6415.691113] LNet: Removed LNI 192.168.203.142@tcp [ 6416.696490] Key type .llcrypt unregistered [ 6416.699395] Key type ._llcrypt unregistered [ 6437.746216] Key type ._llcrypt registered [ 6437.751942] Key type .llcrypt registered [ 6437.909834] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6438.691242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6438.760181] alg: No test for adler32 (adler32-zlib) [ 6439.992904] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 6440.472565] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6442.184223] Key type lgssc registered [ 6443.727434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6453.927336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6453.948869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6455.562087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6462.311211] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6466.028515] LustreError: 219261:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6466.059114] LustreError: 219261:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6471.154459] LustreError: 219260:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6472.785910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6473.456270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6477.953459] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6487.325649] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6494.020365] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6501.768906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6502.231301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6509.506875] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6518.300469] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6525.804129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6537.852635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6537.957530] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6538.268781] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6544.739961] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6553.253896] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6554.964527] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6566.942717] Lustre: Permanently deactivating lustre-OST0001 [ 6566.965263] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6576.177058] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6582.544312] Lustre: Permanently reactivating lustre-OST0001 [ 6582.578372] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6582.586565] Lustre: Skipped 2 previous similar messages [ 6585.607058] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.632540] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6585.648526] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6585.667145] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6585.751710] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6596.432382] Lustre: server umount lustre-OST0001 complete [ 6599.147501] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.162352] Lustre: Skipped 1 previous similar message [ 6599.171257] LustreError: 221769:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6599.189761] LustreError: 221769:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6600.673623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6600.686589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6600.713128] Lustre: Skipped 1 previous similar message [ 6600.731113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6602.726244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6602.744095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6602.771829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6604.258786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6604.261078] LustreError: 221085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6604.281195] Lustre: Skipped 1 previous similar message [ 6606.695372] Lustre: server umount lustre-OST0000 complete [ 6611.425356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6611.430062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.444299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6617.453959] Lustre: server umount lustre-MDT0000 complete [ 6619.618446] LustreError: 219260:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6619.647028] LustreError: 219260:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6621.336020] LustreError: 219242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740979 with bad export cookie 8178371709660284205 [ 6621.337310] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6621.354210] LustreError: 219242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6622.083058] Lustre: server umount lustre-MDT0001 complete [ 6633.662468] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6636.719295] Key type lgssc unregistered [ 6636.998651] LNet: 224367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6637.006312] LNetError: 224367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6637.029501] LNet: Removed LNI 192.168.203.142@tcp [ 6637.834509] Key type .llcrypt unregistered [ 6637.836859] Key type ._llcrypt unregistered [ 6675.567899] Key type ._llcrypt registered [ 6675.573692] Key type .llcrypt registered [ 6675.732707] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6676.600285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6676.690055] alg: No test for adler32 (adler32-zlib) [ 6677.705460] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 6677.963281] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6679.656763] Key type lgssc registered [ 6680.860762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6689.223855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6689.261994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.618558] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6690.649963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6690.851611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6696.339978] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6707.491947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6707.561221] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6707.585230] Lustre: 226585:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6707.876553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6711.391750] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6719.324709] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6724.380357] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6730.918185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6731.141892] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6731.357639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6737.616448] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6739.695950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6745.238408] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6748.905861] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6757.887547] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6765.686960] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6775.484196] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6791.359781] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6791.367440] Lustre: Skipped 1 previous similar message [ 6812.023715] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6812.029692] Lustre: Skipped 1 previous similar message [ 6850.030051] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6850.036257] Lustre: Skipped 4 previous similar messages [ 6926.042473] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6926.049151] Lustre: Skipped 8 previous similar messages [ 6933.989776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6933.993639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6933.999973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6934.500958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6934.510122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6934.530638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6934.530647] Lustre: Skipped 1 previous similar message [ 6938.760959] Lustre: server umount lustre-OST0000 complete [ 6943.713818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6943.717677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6943.728056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6948.820528] Lustre: server umount lustre-MDT0000 complete [ 6949.860839] LustreError: 228869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6949.889836] LustreError: 228869:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6953.095601] LustreError: 226578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741311 with bad export cookie 4614172014772622793 [ 6953.097257] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6953.113775] LustreError: 226578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6953.420353] Lustre: server umount lustre-MDT0001 complete [ 6963.352370] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6966.341105] Key type lgssc unregistered [ 6966.709815] LNet: 231424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6966.716506] LNetError: 231424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6966.731655] LNet: Removed LNI 192.168.203.142@tcp [ 6967.640717] Key type .llcrypt unregistered [ 6967.643057] Key type ._llcrypt unregistered [ 6992.357375] Key type ._llcrypt registered [ 6992.360269] Key type .llcrypt registered [ 6992.489430] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6993.396840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6993.462841] alg: No test for adler32 (adler32-zlib) [ 6994.551849] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 6994.779214] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6996.456221] Key type lgssc registered [ 6997.692612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7005.802858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7005.835655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7007.266902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7011.572273] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7017.442648] LustreError: 232853:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7017.468881] LustreError: 232853:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7020.298350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7020.735208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7025.433200] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7032.553277] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7037.430992] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7043.570794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7043.863243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7052.103198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7052.161546] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7059.899537] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7076.833683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7076.844064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.868322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7077.860202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7077.864427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.879854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7078.250595] Lustre: server umount lustre-OST0000 complete [ 7081.953328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7081.960265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7081.984038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7081.996347] Lustre: Skipped 1 previous similar message [ 7087.500513] Lustre: server umount lustre-MDT0000 complete [ 7088.098218] LustreError: 232853:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.113989] LustreError: 232853:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7090.543065] LustreError: 232834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741448 with bad export cookie 10801116836301806029 [ 7090.550770] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7090.559819] LustreError: 232834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7091.070816] Lustre: server umount lustre-MDT0001 complete [ 7101.056623] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7103.532439] Key type lgssc unregistered [ 7103.770674] LNet: 236539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7103.785418] LNetError: 236539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7103.799802] LNet: Removed LNI 192.168.203.142@tcp [ 7104.749379] Key type .llcrypt unregistered [ 7104.758853] Key type ._llcrypt unregistered [ 7125.672197] Key type ._llcrypt registered [ 7125.673859] Key type .llcrypt registered [ 7125.784556] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7126.347366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7126.389325] alg: No test for adler32 (adler32-zlib) [ 7127.400331] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 7127.636796] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 7129.312417] Key type lgssc registered [ 7130.408891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7402.741290] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7405.481441] Key type lgssc unregistered [ 7405.755844] LNet: 238595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7405.770305] LNetError: 238595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7405.786439] LNet: Removed LNI 192.168.203.142@tcp [ 7406.524504] Key type .llcrypt unregistered [ 7406.527819] Key type ._llcrypt unregistered [ 7431.216370] Key type ._llcrypt registered [ 7431.218951] Key type .llcrypt registered [ 7431.353806] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 7448.954863] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7450.305293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7450.371759] alg: No test for adler32 (adler32-zlib) [ 7451.457517] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 7451.679413] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 7453.360791] Key type lgssc registered [ 7454.766206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7484.313293] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7199 sec ========== 00:17:21 (1790741841) [ 7486.049453] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:17:22 (1790741842) === [ 7489.098810] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:17:26 (1790741846) === [ 7526.742629] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7529.603679] Key type lgssc unregistered [ 7529.930996] LNet: 243843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7529.941658] LNetError: 243843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7529.975241] LNet: Removed LNI 192.168.203.142@tcp [ 7531.004843] Key type .llcrypt unregistered [ 7531.009167] Key type ._llcrypt unregistered