[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 582028294 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003321] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011285] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.014038] *** VALIDATE selinux *** [ 0.024000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029773] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030179] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032114] *** VALIDATE tmpfs *** [ 0.034348] *** VALIDATE proc *** [ 0.035213] *** VALIDATE cgroup *** [ 0.036007] *** VALIDATE cgroup2 *** [ 0.038233] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041032] Spectre V2 : User space: Vulnerable [ 0.043004] Speculative Store Bypass: Vulnerable [ 0.045814] debug: unmapping init [mem 0xffffffffa7a59000-0xffffffffa7a60fff] [ 0.048208] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049832] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050029] ... version: 2 [ 0.051015] ... bit width: 48 [ 0.052013] ... generic registers: 4 [ 0.053012] ... value mask: 0000ffffffffffff [ 0.054016] ... max period: 00007fffffffffff [ 0.055017] ... fixed-purpose events: 3 [ 0.056013] ... event mask: 000000070000000f [ 0.057312] rcu: Hierarchical SRCU implementation. [ 0.059597] smp: Bringing up secondary CPUs ... [ 0.060596] x86: Booting SMP configuration: [ 0.061027] .... node #0, CPUs: #1 #2 #3 [ 0.065293] smp: Brought up 1 node, 4 CPUs [ 0.067015] smpboot: Max logical packages: 1 [ 0.068012] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.149253] node 0 deferred pages initialised in 77ms [ 0.152014] devtmpfs: initialized [ 0.153240] x86/mm: Memory block size: 128MB [ 0.156650] gcov: version magic: 0x41383552 [ 0.157659] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164285] pinctrl core: initialized pinctrl subsystem [ 0.166306] [ 0.166643] ************************************************************* [ 0.168011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170013] ** ** [ 0.173009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175008] ** ** [ 0.178012] ** This means that this kernel is built to expose internal ** [ 0.180009] ** IOMMU data structures, which may compromise security on ** [ 0.182011] ** your system. ** [ 0.184009] ** ** [ 0.185012] ** If you see this message and you are not debugging the ** [ 0.187012] ** kernel, report this immediately to your vendor! ** [ 0.189010] ** ** [ 0.192014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195015] ************************************************************* [ 0.198704] NET: Registered protocol family 16 [ 0.200464] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.204065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209121] cpuidle: using governor menu [ 0.212145] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.213756] PCI: Using configuration type 1 for base access [ 0.214133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.223056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.227114] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231256] cryptd: max_cpu_qlen set to 1000 [ 0.236258] ACPI: Added _OSI(Module Device) [ 0.239018] ACPI: Added _OSI(Processor Device) [ 0.241017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.244021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.251493] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.260371] ACPI: Interpreter enabled [ 0.262066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.264014] ACPI: Using IOAPIC for interrupt routing [ 0.265101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.268548] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.279265] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.282052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.285023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.294477] acpiphp: Slot [2] registered [ 0.297146] acpiphp: Slot [5] registered [ 0.299313] acpiphp: Slot [6] registered [ 0.300105] acpiphp: Slot [7] registered [ 0.302333] acpiphp: Slot [8] registered [ 0.304103] acpiphp: Slot [9] registered [ 0.306276] acpiphp: Slot [10] registered [ 0.307277] acpiphp: Slot [3] registered [ 0.309112] acpiphp: Slot [4] registered [ 0.311112] acpiphp: Slot [11] registered [ 0.313097] acpiphp: Slot [12] registered [ 0.315099] acpiphp: Slot [13] registered [ 0.316126] acpiphp: Slot [14] registered [ 0.318085] acpiphp: Slot [15] registered [ 0.319082] acpiphp: Slot [16] registered [ 0.321096] acpiphp: Slot [17] registered [ 0.322091] acpiphp: Slot [18] registered [ 0.324061] acpiphp: Slot [19] registered [ 0.325000] acpiphp: Slot [20] registered [ 0.327110] acpiphp: Slot [21] registered [ 0.329121] acpiphp: Slot [22] registered [ 0.330085] acpiphp: Slot [23] registered [ 0.332092] acpiphp: Slot [24] registered [ 0.333062] acpiphp: Slot [25] registered [ 0.333934] acpiphp: Slot [26] registered [ 0.335056] acpiphp: Slot [27] registered [ 0.337090] acpiphp: Slot [28] registered [ 0.339350] acpiphp: Slot [29] registered [ 0.342104] acpiphp: Slot [30] registered [ 0.343068] acpiphp: Slot [31] registered [ 0.345075] PCI host bridge to bus 0000:00 [ 0.346012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.348014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.350016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.352019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.355026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.358021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.360144] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.363069] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.367414] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.378016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.382295] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.384016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.387019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.389013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.392126] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.394791] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.397043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.401097] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.408018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.423018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.430017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.435909] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.446015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.452013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.466018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.479484] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.489013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.497017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.520021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.539352] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.546017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.555032] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.574021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.585044] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.598020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.608014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.625026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.641106] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.648015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.654015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.673028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.684786] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.703020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.712025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.737019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.750689] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.753489] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.756587] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.759455] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.762228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.768030] iommu: Default domain type: Passthrough [ 0.770614] SCSI subsystem initialized [ 0.772133] ACPI: bus type USB registered [ 0.773127] usbcore: registered new interface driver usbfs [ 0.775074] usbcore: registered new interface driver hub [ 0.777059] usbcore: registered new device driver usb [ 0.778137] pps_core: LinuxPPS API ver. 1 registered [ 0.780010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.783077] PTP clock support registered [ 0.785123] EDAC MC: Ver: 3.0.0 [ 0.787260] PCI: Using ACPI for IRQ routing [ 0.789914] NetLabel: Initializing [ 0.791012] NetLabel: domain hash size = 128 [ 0.792008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.794069] NetLabel: unlabeled traffic allowed by default [ 0.797134] vgaarb: loaded [ 0.799254] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.801017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.807248] clocksource: Switched to clocksource kvm-clock [ 0.929514] VFS: Disk quotas dquot_6.6.0 [ 0.931387] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.934801] *** VALIDATE ramfs *** [ 0.936561] *** VALIDATE hugetlbfs *** [ 0.938169] pnp: PnP ACPI init [ 0.940872] pnp: PnP ACPI: found 6 devices [ 0.955301] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.959774] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.963079] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.966096] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.969155] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.970756] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.973198] NET: Registered protocol family 2 [ 0.975661] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.979596] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.982188] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.988138] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.991906] TCP: Hash tables configured (established 65536 bind 65536) [ 0.995757] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.999935] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.003768] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.007770] NET: Registered protocol family 1 [ 1.011137] RPC: Registered named UNIX socket transport module. [ 1.014072] RPC: Registered udp transport module. [ 1.016505] RPC: Registered tcp transport module. [ 1.018729] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.021808] NET: Registered protocol family 44 [ 1.023945] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.026802] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.029776] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.034061] PCI: CLS 0 bytes, default 64 [ 1.036351] Unpacking initramfs... [ 2.709904] debug: unmapping init [mem 0xffff8ff8bcc54000-0xffff8ff8bffbffff] [ 2.714464] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.716914] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.720939] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 3.395445] Initialise system trusted keyrings [ 3.397697] Key type blacklist registered [ 3.400590] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.413535] zbud: loaded [ 3.417302] *** VALIDATE nfs *** [ 3.418642] *** VALIDATE nfs4 *** [ 3.420684] pstore: using deflate compression [ 3.425156] Platform Keyring initialized [ 3.555837] NET: Registered protocol family 38 [ 3.558124] Key type asymmetric registered [ 3.560373] Asymmetric key parser 'x509' registered [ 3.563475] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.567066] io scheduler mq-deadline registered [ 3.568995] io scheduler kyber registered [ 3.570787] io scheduler bfq registered [ 3.573084] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.577218] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.580626] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.584307] ACPI: Power Button [PWRF] [ 3.591537] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.599811] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.610476] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.615921] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.641293] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.684645] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.721026] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.728917] Non-volatile memory driver v1.3 [ 3.730140] Linux agpgart interface v0.103 [ 3.768381] virtio_blk virtio1: [vda] 134592 512-byte logical blocks (68.9 MB/65.7 MiB) [ 3.771926] vda: detected capacity change from 0 to 68911104 [ 3.790319] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.793933] vdb: detected capacity change from 0 to 1073741824 [ 3.815873] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.819538] vdc: detected capacity change from 0 to 2621440000 [ 3.841679] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.845656] vdd: detected capacity change from 0 to 2621440000 [ 3.866347] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.870984] vde: detected capacity change from 0 to 4294967296 [ 3.901402] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.905882] vdf: detected capacity change from 0 to 4294967296 [ 3.917643] libphy: Fixed MDIO Bus: probed [ 3.924713] usbcore: registered new interface driver usbserial_generic [ 3.929194] usbserial: USB Serial support registered for generic [ 3.932488] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.938769] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.940736] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.942811] mousedev: PS/2 mouse device common for all mice [ 3.946369] rtc_cmos 00:05: RTC can wake from S4 [ 3.949125] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.954867] rtc_cmos 00:05: registered as rtc0 [ 3.959107] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.960428] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.966266] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.967472] intel_pstate: CPU model not supported [ 3.974906] hid: raw HID events driver (C) Jiri Kosina [ 3.977757] usbcore: registered new interface driver usbhid [ 3.980490] usbhid: USB HID core driver [ 3.982478] drop_monitor: Initializing network drop monitor service [ 3.985528] Initializing XFRM netlink socket [ 3.987710] NET: Registered protocol family 10 [ 3.991554] Segment Routing with IPv6 [ 3.993029] NET: Registered protocol family 17 [ 3.996476] mpls_gso: MPLS GSO support [ 4.004331] RAS: Correctable Errors collector initialized. [ 4.006386] AVX version of gcm_enc/dec engaged. [ 4.007634] AES CTR mode by8 optimization enabled [ 4.114152] sched_clock: Marking stable (4114078970, 0)->(5257501269, -1143422299) [ 4.119161] registered taskstats version 1 [ 4.122480] Loading compiled-in X.509 certificates [ 4.125996] zswap: loaded using pool lzo/zbud [ 4.160073] Key type big_key registered [ 4.175618] Key type encrypted registered [ 4.177142] ima: No TPM chip found, activating TPM-bypass! [ 4.178946] ima: Allocated hash algorithm: sha1 [ 4.180043] ima: No architecture policies found [ 4.181513] evm: Initialising EVM extended attributes: [ 4.184470] evm: security.selinux [ 4.186186] evm: security.ima [ 4.187833] evm: security.capability [ 4.189735] evm: HMAC attrs: 0x1 [ 4.192770] rtc_cmos 00:05: setting system clock to 2026-03-15 20:04:44 UTC (1773605084) [ 4.201629] debug: unmapping init [mem 0xffffffffa8a03000-0xffffffffa8bfffff] [ 4.206098] debug: unmapping init [mem 0xffffffffa7782000-0xffffffffa7a58fff] [ 4.216163] Write protecting the kernel read-only data: 28672k [ 4.221453] debug: unmapping init [mem 0xffffffffa5e03000-0xffffffffa5ffffff] [ 4.226431] debug: unmapping init [mem 0xffffffffa6714000-0xffffffffa67fffff] [ 4.286968] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.294379] systemd[1]: Detected virtualization kvm. [ 4.296580] systemd[1]: Detected architecture x86-64. [ 4.299696] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.336462] systemd[1]: No hostname configured. [ 4.339526] systemd[1]: Set hostname to . [ 4.343357] random: systemd: uninitialized urandom read (16 bytes read) [ 4.346631] systemd[1]: Initializing machine ID from random generator. [ 4.410980] random: ln: uninitialized urandom read (6 bytes read) [ 4.514488] random: systemd: uninitialized urandom read (16 bytes read) [ 4.518337] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.523552] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.530669] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.307691] device-mapper: uevent: version 1.0.3 [ 5.311147] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.329669] virtio_net virtio0 ens2: renamed from eth0 [ 6.334967] random: fast init done [ 6.433558] scsi host0: ata_piix [ 6.457611] scsi host1: ata_piix [ 6.462860] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.466618] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.456760] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.696457] random: crng init done [ 10.697755] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.862165] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.319632] printk: systemd: 25 output lines suppressed due to ratelimiting [ 13.652551] SELinux: Disabled at runtime. [ 13.720826] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.729794] systemd[1]: Detected virtualization kvm. [ 13.732371] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.327456] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.331370] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.335907] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.341390] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.343806] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.352391] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.358496] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 14.456057] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 15.633718] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.094849] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.307134] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.916072] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.019670] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 24.310432] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit)[ 24.997741] NFS: Registering the id_resolver key type [ 25.000133] Key type id_resolver registered [ 25.002144] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. [ 36.848246] hrtimer: interrupt took 3759509 ns Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg120-server login: [ 112.347624] libcfs: loading out-of-tree module taints kernel. [ 112.457984] Key type ._llcrypt registered [ 112.462927] Key type .llcrypt registered [ 112.579744] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 140.532552] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 142.907963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 142.932238] alg: No test for adler32 (adler32-zlib) [ 144.737301] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 146.158764] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 148.072253] Key type lgssc registered [ 150.532168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 180.814674] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 240.728837] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 263.097077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 263.157746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 264.613361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 264.691995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 264.892843] Lustre: lustre-MDT0000: new disk, initializing [ 265.102970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 265.148884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 271.960679] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 292.281666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 292.467160] Lustre: 6513:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 292.533608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 292.545569] Lustre: Skipped 1 previous similar message [ 292.809177] Lustre: lustre-MDT0001: new disk, initializing [ 292.954375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 293.004631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 293.018571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 299.875240] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 306.972280] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 321.330941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 321.809594] Lustre: lustre-OST0000: new disk, initializing [ 321.815139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 321.911877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 329.335858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 329.357064] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 329.559384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 332.596904] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 353.865849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 354.108246] Lustre: lustre-OST0001: new disk, initializing [ 354.114537] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 354.194669] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 359.468917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 359.478941] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 359.525087] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 364.785726] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 381.519953] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 392.478377] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 409.159393] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing check_logdir /tmp/testlogs/ [ 416.525990] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing yml_node [ 423.751080] Lustre: DEBUG MARKER: Client: 2.17.51.2 [ 427.768709] Lustre: DEBUG MARKER: MDS: 2.17.51.2 [ 432.298753] Lustre: DEBUG MARKER: OSS: 2.17.51.2 [ 435.275687] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Mar 15 16:11:53 EDT 2026 [ 460.267105] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 462.562762] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 482.272888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 482.278336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.299921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 482.314804] Lustre: Skipped 1 previous similar message [ 482.330915] Lustre: Skipped 3 previous similar messages [ 487.394870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 487.413937] Lustre: Skipped 3 previous similar messages [ 492.514740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 492.520810] Lustre: Skipped 2 previous similar messages [ 496.096158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 496.370595] Lustre: server umount lustre-MDT0000 complete [ 501.511927] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773605581 with bad export cookie 12903142051599324221 [ 501.512673] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 501.525973] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 502.752983] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.758900] LustreError: 6522:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.759300] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 502.790748] Lustre: Skipped 3 previous similar messages [ 502.838717] LustreError: 6522:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 506.785251] LustreError: 6523:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.786290] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 506.821361] LustreError: 6523:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 506.840419] Lustre: Skipped 2 previous similar messages [ 507.960252] Lustre: server umount lustre-MDT0001 complete [ 513.474302] Lustre: server umount lustre-OST0000 complete [ 519.512581] Lustre: server umount lustre-OST0001 complete [ 529.760686] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 544.383083] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 600.801810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.250874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 601.345380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 601.509554] Lustre: lustre-MDT0000: new disk, initializing [ 601.681888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.711769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 608.589823] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 626.243963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.487742] Lustre: 15018:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 626.651528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 626.658360] Lustre: Skipped 1 previous similar message [ 626.746209] Lustre: lustre-MDT0001: new disk, initializing [ 626.949669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 627.053538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 627.083605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 633.606991] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 644.573163] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 649.905989] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 659.763520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 660.270345] Lustre: lustre-OST0000: new disk, initializing [ 660.276979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 660.440518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.143976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 662.151582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 662.240947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 669.014742] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 681.293648] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 689.556112] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 689.979878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 697.097992] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 697.552556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 703.484662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 703.506368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.533367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 703.544092] Lustre: Skipped 1 previous similar message [ 706.963987] Lustre: server umount lustre-OST0000 complete [ 713.702823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.715518] Lustre: Skipped 1 previous similar message [ 719.686342] Lustre: server umount lustre-MDT0000 complete [ 722.929644] LustreError: 15028:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.955202] LustreError: 15028:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 724.947355] LustreError: 15010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773605805 with bad export cookie 12903142051599325761 [ 724.952409] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 724.969940] LustreError: 15010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 725.376265] Lustre: server umount lustre-MDT0001 complete [ 730.759582] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 16:16:49 (1773605809) [ 741.028916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 741.971059] LustreError: 18885:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 742.171555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 747.500393] LustreError: 18885:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 747.519326] LustreError: 18885:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 748.824768] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 757.731108] LustreError: 18886:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 757.749502] LustreError: 18886:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 762.199832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 769.632438] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 779.071966] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 785.896128] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 799.079410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 811.166101] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 820.542404] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 832.484111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 832.519534] Lustre: Skipped 1 previous similar message [ 832.540407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 832.560224] Lustre: Skipped 5 previous similar messages [ 833.961649] Lustre: server umount lustre-OST0000 complete [ 842.724896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 842.745105] Lustre: Skipped 1 previous similar message [ 846.272336] Lustre: server umount lustre-MDT0000 complete [ 847.848773] LustreError: 18885:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 847.886361] LustreError: 18885:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 851.315448] LustreError: 19605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773605931 with bad export cookie 12903142051599326643 [ 851.321330] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 851.336466] LustreError: 19605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 851.801324] Lustre: server umount lustre-MDT0001 complete [ 867.388258] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 871.290199] Key type lgssc unregistered [ 871.793570] LNet: 22073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.800933] LNetError: 22073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.819424] LNet: Removed LNI 192.168.201.120@tcp [ 873.202249] Key type .llcrypt unregistered [ 873.205451] Key type ._llcrypt unregistered [ 906.932833] Key type ._llcrypt registered [ 906.935274] Key type .llcrypt registered [ 907.045645] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 908.893390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 908.938627] alg: No test for adler32 (adler32-zlib) [ 910.395613] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 910.810339] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 912.560324] Key type lgssc registered [ 914.631892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 926.861744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 926.929758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.632029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 934.369985] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 937.959217] LustreError: 23474:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 938.006295] LustreError: 23474:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 943.074723] LustreError: 23473:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 947.456527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 948.452372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 955.159587] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 966.812817] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 972.650895] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 983.321749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 983.919904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.630769] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 998.305490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1001.506732] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1023.970833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.982376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1024.003682] Lustre: Skipped 1 previous similar message [ 1024.026298] Lustre: Skipped 1 previous similar message [ 1028.879980] Lustre: server umount lustre-OST0000 complete [ 1035.234983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1035.246320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.278822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1039.338670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1039.346151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.953425] Lustre: server umount lustre-MDT0000 complete [ 1044.455870] LustreError: 23468:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1044.494504] LustreError: 23468:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1046.463973] LustreError: 23453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773606126 with bad export cookie 2891009561305925585 [ 1046.468929] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1046.477951] LustreError: 23453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1047.015559] Lustre: server umount lustre-MDT0001 complete [ 1067.268964] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1071.320662] Key type lgssc unregistered [ 1071.850954] LNet: 27125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.868164] LNetError: 27125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.920779] LNet: Removed LNI 192.168.201.120@tcp [ 1073.034579] Key type .llcrypt unregistered [ 1073.037991] Key type ._llcrypt unregistered [ 1108.023980] Key type ._llcrypt registered [ 1108.027772] Key type .llcrypt registered [ 1108.184107] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1109.988022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1110.192087] alg: No test for adler32 (adler32-zlib) [ 1111.537494] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 1111.783622] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1113.562161] Key type lgssc registered [ 1115.573244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1128.545293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1128.577917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1130.240829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1137.079622] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1140.706780] LustreError: 28526:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1140.723262] LustreError: 28526:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1145.833269] LustreError: 28525:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.548427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.140390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1156.289487] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1165.831198] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1171.050901] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1190.746595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1191.344277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.888559] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1201.642203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1210.012028] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1222.122086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1222.124947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1222.139644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1222.170680] Lustre: Skipped 1 previous similar message [ 1223.591593] Lustre: server umount lustre-OST0000 complete [ 1231.845625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1231.857558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.885046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1231.892758] Lustre: Skipped 1 previous similar message [ 1236.253437] Lustre: server umount lustre-MDT0000 complete [ 1237.474737] LustreError: 28525:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.520664] LustreError: 28525:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1241.727858] LustreError: 28508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773606322 with bad export cookie 15181157250463173052 [ 1241.731427] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.737386] LustreError: 28508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1242.139654] Lustre: server umount lustre-MDT0001 complete [ 1263.264549] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1268.259700] Key type lgssc unregistered [ 1268.728654] LNet: 32076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.738405] LNetError: 32076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.763923] LNet: Removed LNI 192.168.201.120@tcp [ 1270.236166] Key type .llcrypt unregistered [ 1270.239121] Key type ._llcrypt unregistered [ 1303.742863] Key type ._llcrypt registered [ 1303.744506] Key type .llcrypt registered [ 1303.945650] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1306.638205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1306.661992] alg: No test for adler32 (adler32-zlib) [ 1308.093861] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 1308.474117] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1310.312339] Key type lgssc registered [ 1312.051215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1324.639914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1324.676839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1326.569259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.868350] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1335.780787] LustreError: 33476:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.814637] LustreError: 33476:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1340.915325] LustreError: 33475:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.020471] LustreError: 33476:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.882627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.661354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1354.511448] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1363.693250] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1370.233892] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1379.262829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1379.888581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.964230] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1396.298566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1402.683378] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1421.321512] Lustre: server umount lustre-OST0000 complete [ 1429.475538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1429.498729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.531176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1432.043533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.054255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1432.094718] Lustre: Skipped 1 previous similar message [ 1437.154449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1437.168194] Lustre: Skipped 1 previous similar message [ 1438.176668] Lustre: 32784:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773606502/real 1773606502] req@0000000044078477 x1859761091939200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773606518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1438.257880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.282069] LustreError: 33476:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.325941] LustreError: 33476:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1438.491485] Lustre: server umount lustre-MDT0000 complete [ 1439.329906] Lustre: 32784:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773606503/real 1773606503] req@0000000018c12e93 x1859761091939712/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773606519 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1443.296140] Lustre: 32786:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773606507/real 1773606507] req@00000000ea58f94e x1859761091940608/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773606523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1443.305670] LustreError: 34587:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.382512] LustreError: 34587:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1445.951368] LustreError: 33456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773606526 with bad export cookie 9744598808978642383 [ 1445.969071] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1445.973683] LustreError: 33456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1447.216922] Lustre: server umount lustre-MDT0001 complete [ 1468.001687] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1473.023733] Key type lgssc unregistered [ 1473.575912] LNet: 36702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.591738] LNetError: 36702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1473.609619] LNet: Removed LNI 192.168.201.120@tcp [ 1474.985330] Key type .llcrypt unregistered [ 1475.005488] Key type ._llcrypt unregistered [ 1510.990593] Key type ._llcrypt registered [ 1511.002410] Key type .llcrypt registered [ 1511.240529] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1513.734413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1513.796435] alg: No test for adler32 (adler32-zlib) [ 1515.105756] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 1515.657650] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1517.440561] Key type lgssc registered [ 1520.388882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1536.526316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1536.561761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.726152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.941040] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1548.265865] LustreError: 38101:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.301618] LustreError: 38101:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1553.379933] LustreError: 38100:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.513922] LustreError: 38101:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.877336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1559.804385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1567.998728] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1578.732502] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1584.670934] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1596.800217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1597.797227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.103047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1608.832539] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1618.482112] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1628.648527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1628.650793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.675726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1628.743479] Lustre: Skipped 1 previous similar message [ 1632.413790] Lustre: lustre-OST0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 1632.428723] Lustre: Skipped 1 previous similar message [ 1633.763055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1633.775466] Lustre: Skipped 1 previous similar message [ 1634.433768] Lustre: server umount lustre-OST0000 complete [ 1661.929777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1661.942787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.973726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1664.501472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.286460] Lustre: server umount lustre-MDT0000 complete [ 1669.607400] LustreError: 38101:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.625079] LustreError: 38101:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1672.804581] LustreError: 38081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773606753 with bad export cookie 17583303041953293770 [ 1672.827971] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1679.442584] Lustre: server umount lustre-MDT0001 complete [ 1699.886747] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1704.802672] Key type lgssc unregistered [ 1705.230609] LNet: 41483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.248860] LNetError: 41483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.273207] LNet: Removed LNI 192.168.201.120@tcp [ 1706.757622] Key type .llcrypt unregistered [ 1706.762671] Key type ._llcrypt unregistered [ 1740.778751] Key type ._llcrypt registered [ 1740.789726] Key type .llcrypt registered [ 1740.977229] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1742.884518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1742.920638] alg: No test for adler32 (adler32-zlib) [ 1744.336367] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 1744.856210] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1746.624157] Key type lgssc registered [ 1748.258420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1759.975519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1759.992449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1761.633643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.298962] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1771.001115] LustreError: 42879:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1771.051938] LustreError: 42879:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1776.107889] LustreError: 42881:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1781.232720] LustreError: 42879:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1784.008054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.772322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1792.672719] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1801.701588] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1807.898599] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1819.831882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1820.686156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.895414] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1830.934621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1840.618477] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1851.375549] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.380923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1851.417333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1853.064657] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 1853.070553] Lustre: Skipped 2 previous similar messages [ 1856.043082] Lustre: server umount lustre-MDT0000 complete [ 1856.488361] LustreError: 42875:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1856.514309] LustreError: 42875:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1861.104629] LustreError: 42860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773606941 with bad export cookie 16217889309003587048 [ 1861.120988] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1861.142441] LustreError: 42860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1861.601722] LustreError: 42875:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1861.620620] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.635189] LustreError: 42875:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1861.691252] Lustre: Skipped 2 previous similar messages [ 1861.706875] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1866.726846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1866.742791] Lustre: Skipped 1 previous similar message [ 1867.679076] Lustre: server umount lustre-MDT0001 complete [ 1884.999446] Lustre: server umount lustre-OST0000 complete [ 1910.966554] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1916.042644] Key type lgssc unregistered [ 1916.546407] LNet: 46395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.563882] LNetError: 46395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.586181] LNet: Removed LNI 192.168.201.120@tcp [ 1917.789537] Key type .llcrypt unregistered [ 1917.793963] Key type ._llcrypt unregistered [ 1956.025173] Key type ._llcrypt registered [ 1956.029783] Key type .llcrypt registered [ 1956.288824] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1957.956871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1957.993105] alg: No test for adler32 (adler32-zlib) [ 1959.386141] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 1959.682450] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1961.352185] Key type lgssc registered [ 1963.132686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1976.902400] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1976.938302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1992.740563] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1997.856515] LustreError: 47768:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 2024.737975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.425366] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2044.476530] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2212.463304] Lustre: server umount lustre-OST0000 complete [ 2243.657799] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2249.482439] Key type lgssc unregistered [ 2250.124088] LNet: 49344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.144482] LNetError: 49344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2250.177979] LNet: Removed LNI 192.168.201.120@tcp [ 2252.362954] Key type .llcrypt unregistered [ 2252.369589] Key type ._llcrypt unregistered [ 2292.226965] Key type ._llcrypt registered [ 2292.231894] Key type .llcrypt registered [ 2292.379367] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2293.443957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2293.528972] alg: No test for adler32 (adler32-zlib) [ 2294.787926] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 2295.283476] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2297.106543] Key type lgssc registered [ 2298.727631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2311.371807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2311.442661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.582059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2320.536742] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2322.927551] LustreError: 50743:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.959857] LustreError: 50743:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2328.048602] LustreError: 50744:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.158488] LustreError: 50743:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.925859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2334.507091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2341.648157] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2350.591466] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2357.378084] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2369.072635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2369.758869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2379.656691] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2383.104743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2388.597460] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2393.569653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2393.580248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.582910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2393.616888] Lustre: Skipped 1 previous similar message [ 2397.864298] Lustre: server umount lustre-OST0000 complete [ 2403.819050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2403.841756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2403.865181] Lustre: Skipped 2 previous similar messages [ 2408.929708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2408.949679] Lustre: Skipped 1 previous similar message [ 2410.064405] Lustre: server umount lustre-MDT0000 complete [ 2413.096313] LustreError: 50739:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.124714] LustreError: 50739:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2415.811809] LustreError: 50725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773607496 with bad export cookie 2876086524558681801 [ 2415.821449] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2415.826375] LustreError: 50725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2416.431902] Lustre: server umount lustre-MDT0001 complete [ 2434.563125] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2438.582988] Key type lgssc unregistered [ 2439.049887] LNet: 53943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.069331] LNetError: 53943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2439.098418] LNet: Removed LNI 192.168.201.120@tcp [ 2440.507592] Key type .llcrypt unregistered [ 2440.510161] Key type ._llcrypt unregistered [ 2476.844734] Key type ._llcrypt registered [ 2476.849664] Key type .llcrypt registered [ 2477.130702] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2478.850635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2478.989495] alg: No test for adler32 (adler32-zlib) [ 2480.221788] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 2480.578199] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2482.385666] Key type lgssc registered [ 2484.580554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2497.824982] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2497.911494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2513.569824] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2518.689429] LustreError: 55317:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 2545.505313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.289326] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2564.140923] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2576.046593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.648385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.778473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2583.691853] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2585.072388] LustreError: 56243:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.124490] LustreError: 56243:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 2590.188524] LustreError: 55343:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.210466] LustreError: 55343:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2595.299848] LustreError: 56241:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.328404] LustreError: 56241:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2596.649208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.533388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2604.809039] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2614.368334] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2621.162164] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2628.066104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2628.078231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2628.123907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2628.142391] Lustre: Skipped 1 previous similar message [ 2631.608416] Lustre: server umount lustre-OST0000 complete [ 2658.987424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.020436] Lustre: Skipped 1 previous similar message [ 2659.031541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2663.909471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2663.918404] LustreError: Skipped 1 previous similar message [ 2663.930913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.966839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2665.370253] LustreError: 57979:0:(ldlm_resource.c:1170:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000014a4e987) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2665.626946] Lustre: server umount lustre-MDT0000 complete [ 2669.554156] LustreError: 56243:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.593038] LustreError: 56243:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2671.793593] LustreError: 55324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773607752 with bad export cookie 9374226887902044816 [ 2671.800053] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2671.811427] LustreError: 55324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2678.395825] Lustre: server umount lustre-MDT0001 complete [ 2695.860320] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2699.807178] Key type lgssc unregistered [ 2700.354867] LNet: 58719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2700.364839] LNetError: 58719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2700.386481] LNet: Removed LNI 192.168.201.120@tcp [ 2701.850595] Key type .llcrypt unregistered [ 2701.862109] Key type ._llcrypt unregistered [ 2740.567101] Key type ._llcrypt registered [ 2740.582482] Key type .llcrypt registered [ 2740.818802] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2743.327911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2743.385128] alg: No test for adler32 (adler32-zlib) [ 2744.715480] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 2745.081868] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2746.832540] Key type lgssc registered [ 2748.629259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2762.097986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2762.118227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.950538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.492130] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2774.498469] LustreError: 60117:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.527769] LustreError: 60117:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2779.634745] LustreError: 60116:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2783.862979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.426143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2791.307349] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2803.289553] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2810.788284] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2822.280255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2822.755908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2833.293791] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2834.083028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2843.175235] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2865.122429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2865.123951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2865.152538] LustreError: Skipped 1 previous similar message [ 2865.154050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2865.207028] Lustre: Skipped 1 previous similar message [ 2868.207269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2868.214907] Lustre: Skipped 2 previous similar messages [ 2870.127609] Lustre: server umount lustre-OST0000 complete [ 2876.388385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2876.408070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.457149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2878.437639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.460320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2878.469033] Lustre: Skipped 1 previous similar message [ 2882.877661] Lustre: server umount lustre-MDT0000 complete [ 2883.559887] LustreError: 60829:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.590964] LustreError: 60829:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2888.676552] LustreError: 60116:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.705553] LustreError: 60116:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2888.822369] LustreError: 60098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773607969 with bad export cookie 13534765610659278274 [ 2888.840976] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2888.848706] LustreError: 60098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2889.318167] Lustre: server umount lustre-MDT0001 complete [ 2905.584810] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2909.617991] Key type lgssc unregistered [ 2910.053093] LNet: 63323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2910.058572] LNetError: 63323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2910.084155] LNet: Removed LNI 192.168.201.120@tcp [ 2911.260775] Key type .llcrypt unregistered [ 2911.264444] Key type ._llcrypt unregistered [ 2952.082490] Key type ._llcrypt registered [ 2952.092554] Key type .llcrypt registered [ 2952.278354] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2954.090304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2954.407745] alg: No test for adler32 (adler32-zlib) [ 2955.667252] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 2956.195276] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2958.179687] Key type lgssc registered [ 2960.163341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2976.003926] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 16:54:14 (1773608054) [ 2987.563653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2987.601322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2989.845896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.785361] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3000.307534] LustreError: 65015:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.340878] LustreError: 65015:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3005.412281] LustreError: 65014:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3010.536836] LustreError: 65015:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3011.803994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3012.540085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3019.484704] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3028.791534] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3034.517975] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3048.584433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3049.204024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3057.887455] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3060.511876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 3068.116888] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3077.222308] Lustre: Failing over lustre-MDT0000 [ 3077.683659] Lustre: server umount lustre-MDT0000 complete [ 3079.149380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3079.155378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3079.165776] LustreError: 65014:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3081.211497] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3081.239865] Lustre: Skipped 1 previous similar message [ 3086.309637] LustreError: 65726:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.349913] LustreError: 65726:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 3092.532476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.636623] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.794755] Lustre: *** cfs_fail_loc=135, val=0*** [ 3092.955526] LustreError: 67773:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000 failed (-2) [ 3092.963782] LustreError: 67773:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.120@tcp: cfg command failed: rc = -2 [ 3092.981618] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3092.981618] [ 3092.992818] LustreError: MGC192.168.201.120@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3093.008686] LustreError: 67761:0:(tgt_mount.c:1820:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3093.019926] LustreError: 67761:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3093.033740] LustreError: 67761:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-MDT0000 [ 3093.042943] LustreError: 67761:0:(tgt_mount.c:115:server_deregister_mount()) lustre-MDT0000 not registered [ 3093.071542] Lustre: server umount lustre-MDT0000 complete [ 3093.073141] LustreError: 67761:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3096.562623] LustreError: 65009:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.589449] LustreError: 65009:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 3102.780577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.157509] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3103.194354] LustreError: 65007:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3103.311641] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (not set up) [ 3103.745636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3103.889760] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3108.445181] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3108.862754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3108.904447] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3108.982810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 3109.848980] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3124.194438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3124.220528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3124.226205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3126.576667] Lustre: server umount lustre-OST0000 complete [ 3134.434304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.442545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.458105] Lustre: Skipped 2 previous similar messages [ 3134.487617] Lustre: Skipped 2 previous similar messages [ 3138.927720] Lustre: server umount lustre-MDT0000 complete [ 3139.555583] LustreError: 65015:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3139.576351] LustreError: 65015:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 3144.522706] LustreError: 64995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608224 with bad export cookie 12271385359978132446 [ 3144.526582] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3144.535323] LustreError: 64995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3145.080029] Lustre: server umount lustre-MDT0001 complete [ 3165.936974] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 3170.082118] Key type lgssc unregistered [ 3170.523650] LNet: 69545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3170.541719] LNetError: 69545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3170.569934] LNet: Removed LNI 192.168.201.120@tcp [ 3171.879512] Key type .llcrypt unregistered [ 3171.883489] Key type ._llcrypt unregistered [ 3207.347818] Key type ._llcrypt registered [ 3207.352935] Key type .llcrypt registered [ 3207.641038] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3210.479542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3210.500271] alg: No test for adler32 (adler32-zlib) [ 3211.753194] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 3212.359329] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 3214.320521] Key type lgssc registered [ 3216.295408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3228.198248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3228.250135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3229.950591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.147388] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3240.429515] LustreError: 70944:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.463684] LustreError: 70944:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3245.556921] LustreError: 70943:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.628449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.659740] LustreError: 70944:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.518430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3257.707050] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3267.505461] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3274.190297] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3285.113336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3285.835077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3295.512604] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3300.241219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 3305.360360] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3313.986751] Lustre: Failing over lustre-MDT0000 [ 3314.766421] Lustre: server umount lustre-MDT0000 complete [ 3315.692798] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.708583] LustreError: 72249:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3315.730369] Lustre: Skipped 1 previous similar message [ 3315.783184] LustreError: 72249:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3320.808445] LustreError: 70938:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.891182] LustreError: 70938:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3329.121172] LustreError: 70938:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.152323] LustreError: 70938:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 3331.044321] Lustre: 70252:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608395/real 1773608395] req@0000000071865552 x1859763088422528/t0(0) o400->MGC192.168.201.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773608411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3331.100752] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3331.661905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.293937] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1da2ea62dc2392b8 [ 3341.309953] Lustre: MGC192.168.201.120@tcp: Connection restored to 0@lo (at 0@lo) [ 3341.829876] Lustre: *** cfs_fail_loc=a05, val=0*** [ 3342.135598] LustreError: 73704:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000 failed (-9) [ 3342.147544] LustreError: 73704:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.120@tcp: cfg command failed: rc = -9 [ 3342.167505] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3342.167505] [ 3342.184348] LustreError: MGC192.168.201.120@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3342.204267] LustreError: 73689:0:(tgt_mount.c:1820:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 3342.217959] LustreError: 73689:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 3342.249974] LustreError: 73689:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-MDT0000 [ 3342.268449] LustreError: 73689:0:(tgt_mount.c:115:server_deregister_mount()) lustre-MDT0000 not registered [ 3342.378050] Lustre: server umount lustre-MDT0000 complete [ 3342.409268] LustreError: 73689:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 3346.415243] LustreError: 70944:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3346.437655] LustreError: 70944:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 14 previous similar messages [ 3354.617062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.923733] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3354.937231] LustreError: 70935:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3355.521130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3355.590815] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3359.834787] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3360.825520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3360.887815] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3360.930479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 3362.018627] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3376.115874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3376.142282] Lustre: Skipped 1 previous similar message [ 3376.150940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3376.167399] Lustre: Skipped 1 previous similar message [ 3379.728419] Lustre: server umount lustre-OST0000 complete [ 3386.339703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.352985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3386.365956] Lustre: Skipped 2 previous similar messages [ 3386.402356] Lustre: Skipped 1 previous similar message [ 3391.465792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3391.468896] Lustre: Skipped 1 previous similar message [ 3392.604397] Lustre: server umount lustre-MDT0000 complete [ 3396.577555] LustreError: 70939:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3396.624127] LustreError: 70939:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 3398.086426] LustreError: 70925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608478 with bad export cookie 2135526883648050009 [ 3398.093318] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3398.109852] LustreError: 70925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3398.988352] Lustre: server umount lustre-MDT0001 complete [ 3416.313191] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 3420.638836] Key type lgssc unregistered [ 3421.177539] LNet: 75472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3421.212127] LNetError: 75472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3421.258480] LNet: Removed LNI 192.168.201.120@tcp [ 3422.771157] Key type .llcrypt unregistered [ 3422.773656] Key type ._llcrypt unregistered [ 3445.250643] Key type ._llcrypt registered [ 3445.253725] Key type .llcrypt registered [ 3445.400110] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 3445.969345] Key type .llcrypt unregistered [ 3445.973157] Key type ._llcrypt unregistered [ 3481.641516] Key type ._llcrypt registered [ 3481.642918] Key type .llcrypt registered [ 3481.784825] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3483.682462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3483.735830] alg: No test for adler32 (adler32-zlib) [ 3485.120390] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 3485.338184] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 3487.032423] Key type lgssc registered [ 3488.892365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3500.464719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3500.514874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.367887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3509.426483] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3511.793062] LustreError: 77347:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3511.848930] LustreError: 77347:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3516.907125] LustreError: 77366:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.026140] LustreError: 77347:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.744879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.364847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3529.833781] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3537.674464] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3544.331838] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3555.777947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3556.590241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3567.116629] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3572.014973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 3577.196422] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3596.856617] Lustre: server umount lustre-OST0000 complete [ 3602.913650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.918173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3602.933520] Lustre: Skipped 1 previous similar message [ 3608.035474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3608.053235] Lustre: Skipped 2 previous similar messages [ 3613.156704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3613.169949] Lustre: Skipped 1 previous similar message [ 3614.176600] Lustre: 76659:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608678/real 1773608678] req@0000000080c6db0d x1859763374687488/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773608694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3614.223686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.200127] Lustre: 76658:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608679/real 1773608679] req@00000000bff40913 x1859763374687744/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773608695 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3617.248151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3617.647329] Lustre: server umount lustre-MDT0000 complete [ 3619.108793] Lustre: 76657:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608683/real 1773608683] req@00000000d875cab3 x1859763374689280/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773608699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3619.301307] LustreError: 78653:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3619.327983] LustreError: 78653:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3623.337303] LustreError: 77328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608703 with bad export cookie 17784070580950404749 [ 3623.342656] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3623.379414] LustreError: 77328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3623.932385] Lustre: server umount lustre-MDT0001 complete [ 3638.821610] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 3642.772702] Key type lgssc unregistered [ 3643.237148] LNet: 80555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3643.269965] LNetError: 80555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3643.297956] LNet: Removed LNI 192.168.201.120@tcp [ 3644.595122] Key type .llcrypt unregistered [ 3644.596638] Key type ._llcrypt unregistered [ 3679.006928] Key type ._llcrypt registered [ 3679.011589] Key type .llcrypt registered [ 3679.194621] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3680.678157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3680.828832] alg: No test for adler32 (adler32-zlib) [ 3682.264852] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 3682.681166] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 3684.609369] Key type lgssc registered [ 3686.574391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3697.984463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3698.004268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3700.108372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3707.095597] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3710.453355] LustreError: 81955:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3710.487235] LustreError: 81955:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3715.558317] LustreError: 81954:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3720.674577] LustreError: 81955:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3723.202832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.094828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3733.256963] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3744.501980] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3752.161196] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3764.200351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3764.644753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3769.797965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3773.936833] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3783.298852] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3794.403168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3794.423154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3794.464432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3795.425233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3795.433034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3795.457443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3795.535904] Lustre: Skipped 1 previous similar message [ 3798.620097] Lustre: server umount lustre-OST0000 complete [ 3805.671674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3805.698413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3805.705765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3805.754336] Lustre: Skipped 1 previous similar message [ 3810.785093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3810.788355] Lustre: Skipped 2 previous similar messages [ 3811.506950] Lustre: server umount lustre-MDT0000 complete [ 3815.911237] LustreError: 81949:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3815.936853] LustreError: 81949:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3817.584410] LustreError: 83578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608897 with bad export cookie 7260546999497007015 [ 3817.592666] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3817.605442] LustreError: 83578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3818.322155] Lustre: server umount lustre-MDT0001 complete [ 3837.933107] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 3842.292488] Key type lgssc unregistered [ 3842.742449] LNet: 85165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3842.751894] LNetError: 85165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3843.824951] LNet: Removed LNI 192.168.201.120@tcp [ 3845.094466] Key type .llcrypt unregistered [ 3845.114427] Key type ._llcrypt unregistered [ 3884.318962] Key type ._llcrypt registered [ 3884.325025] Key type .llcrypt registered [ 3884.480638] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3885.866556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3885.962722] alg: No test for adler32 (adler32-zlib) [ 3888.158426] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 3888.834333] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 3890.865793] Key type lgssc registered [ 3893.608551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3905.977495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3906.010672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.834154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3914.324580] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3918.306209] LustreError: 86561:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.349567] LustreError: 86561:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3923.439323] LustreError: 86562:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.492932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.048242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3935.219675] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3945.177325] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3951.169428] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3960.618476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3961.150489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3970.250134] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3978.636605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3980.746461] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4004.340532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4004.355265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4004.366895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.392487] Lustre: Skipped 1 previous similar message [ 4004.807785] Lustre: server umount lustre-OST0000 complete [ 4014.564587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4014.570719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.585286] Lustre: Skipped 2 previous similar messages [ 4014.623712] Lustre: Skipped 1 previous similar message [ 4017.108453] Lustre: server umount lustre-MDT0000 complete [ 4019.682752] LustreError: 86562:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4019.722549] LustreError: 86562:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4023.078638] LustreError: 86543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773609103 with bad export cookie 9575432028007137852 [ 4023.088823] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.102193] LustreError: 86543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4023.869162] Lustre: server umount lustre-MDT0001 complete [ 4045.986967] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 4050.757103] Key type lgssc unregistered [ 4051.187190] LNet: 89803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4051.207049] LNetError: 89803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4051.239634] LNet: Removed LNI 192.168.201.120@tcp [ 4052.496109] Key type .llcrypt unregistered [ 4052.497747] Key type ._llcrypt unregistered [ 4086.488489] Key type ._llcrypt registered [ 4086.492469] Key type .llcrypt registered [ 4086.606175] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4088.177967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4088.291090] alg: No test for adler32 (adler32-zlib) [ 4089.468421] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 4089.782983] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 4091.536216] Key type lgssc registered [ 4093.477285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4106.592432] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4106.632807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4122.592331] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4127.714908] LustreError: 91177:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 4154.593673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.594489] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4176.215403] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4186.728068] Lustre: server umount lustre-OST0000 complete [ 4202.955991] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 17:14:40 (1773609280) [ 4215.092817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.079680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4221.409430] LustreError: 92705:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.465902] LustreError: 92705:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4224.816730] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4226.533525] LustreError: 92706:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4231.661549] LustreError: 92705:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.771247] LustreError: 92706:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.874447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.904696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4247.415918] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4256.134504] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4263.197804] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4274.002309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4274.602049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4285.112909] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4285.892501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 4294.695955] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4314.606345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4314.626209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.652390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4316.671098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4316.681271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.689077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4318.371566] Lustre: server umount lustre-OST0000 complete [ 4325.857929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4325.871798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.905499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4325.923307] Lustre: Skipped 1 previous similar message [ 4332.002939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4332.013166] Lustre: Skipped 3 previous similar messages [ 4337.122650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4337.127987] Lustre: Skipped 1 previous similar message [ 4339.168160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4339.460054] Lustre: server umount lustre-MDT0000 complete [ 4342.252538] LustreError: 93418:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4342.312650] LustreError: 93418:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4347.367077] LustreError: 92685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773609427 with bad export cookie 9416413994032110847 [ 4347.408102] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4347.414578] LustreError: 92685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4347.997598] Lustre: server umount lustre-MDT0001 complete [ 4370.509136] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 4375.093489] Key type lgssc unregistered [ 4375.898548] LNet: 96072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4375.908207] LNetError: 96072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4375.964221] LNet: Removed LNI 192.168.201.120@tcp [ 4378.290111] Key type .llcrypt unregistered [ 4378.294336] Key type ._llcrypt unregistered [ 4419.135363] Key type ._llcrypt registered [ 4419.137957] Key type .llcrypt registered [ 4419.270176] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4420.955664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4421.119737] alg: No test for adler32 (adler32-zlib) [ 4422.383765] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 4422.743714] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 4424.553047] Key type lgssc registered [ 4427.263778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4442.892547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4442.920620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4444.760657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4451.506923] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4453.863144] LustreError: 97475:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4453.930290] LustreError: 97475:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4458.992607] LustreError: 97476:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.111586] LustreError: 97475:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4467.245170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4468.040047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4475.932384] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4485.858239] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4492.802624] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4504.665252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4505.321105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4514.661860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 4515.641870] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4525.822354] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4544.485045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4544.498956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4544.542160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4545.505883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.509560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4548.787710] Lustre: server umount lustre-OST0000 complete [ 4560.354193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4560.377551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4560.409519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4560.427858] Lustre: Skipped 1 previous similar message [ 4562.526785] Lustre: server umount lustre-MDT0000 complete [ 4565.986876] LustreError: 97476:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4566.012347] LustreError: 97476:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4569.052244] LustreError: 97457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773609649 with bad export cookie 7702646799379546910 [ 4569.072545] LustreError: 97457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4569.076557] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4569.923891] Lustre: server umount lustre-MDT0001 complete [ 4589.368521] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 4594.256858] Key type lgssc unregistered [ 4594.747904] LNet: 100700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4594.787297] LNetError: 100700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4594.819978] LNet: Removed LNI 192.168.201.120@tcp [ 4596.419728] Key type .llcrypt unregistered [ 4596.422096] Key type ._llcrypt unregistered [ 4625.118357] Key type ._llcrypt registered [ 4625.120386] Key type .llcrypt registered [ 4625.285850] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4626.603244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4626.682837] alg: No test for adler32 (adler32-zlib) [ 4628.343935] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 4628.626824] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 4630.337117] Key type lgssc registered [ 4632.090954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4644.518283] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4644.546272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4660.320806] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4665.440876] LustreError: 101926:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 4692.192566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4702.011562] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4711.182287] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4721.188852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.704669] LustreError: MGC192.168.201.120@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4721.724114] LustreError: 102837:0:(tgt_mount.c:1820:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4721.731695] LustreError: 102837:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 4721.746417] LustreError: 102837:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-MDT0000 [ 4721.900289] Lustre: server umount lustre-MDT0000 complete [ 4721.912395] LustreError: 102837:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4743.136337] Lustre: 101263:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773609807/real 1773609807] req@000000000f43010a x1859764573179392/t0(0) o400->MGC192.168.201.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773609823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4743.137522] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4743.164612] Lustre: 101263:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4743.192158] LustreError: 101939:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4763.735978] Lustre: server umount lustre-OST0000 complete [ 4777.781757] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4794.356030] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4858.373582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4858.969876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4859.024232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4859.173650] Lustre: lustre-MDT0000: new disk, initializing [ 4859.532530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4859.593729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4866.686224] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4882.198870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4882.350775] Lustre: 105618:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4882.409230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4882.415945] Lustre: Skipped 1 previous similar message [ 4882.532348] Lustre: lustre-MDT0001: new disk, initializing [ 4882.759047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4882.819345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4882.847995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4889.646798] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4901.299904] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4908.129280] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4918.999622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4919.538927] Lustre: lustre-OST0000: new disk, initializing [ 4919.551828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4919.784696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4928.144595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4928.159548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4928.367349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4929.130988] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4939.448687] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4945.672983] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4945.991147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4953.021107] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4953.468778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4959.207994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4959.210585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4959.236969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4964.335725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4964.354853] Lustre: Skipped 2 previous similar messages [ 4969.453171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4969.472255] Lustre: Skipped 1 previous similar message [ 4970.978557] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4971.326291] Lustre: server umount lustre-OST0000 complete [ 4979.686899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4979.737974] Lustre: Skipped 1 previous similar message [ 4979.749593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4979.764580] Lustre: Skipped 1 previous similar message [ 4982.753462] Lustre: server umount lustre-MDT0000 complete [ 4987.760483] LustreError: 105611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610068 with bad export cookie 2478949963028477801 [ 4987.765698] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4987.772439] LustreError: 105611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4988.341875] Lustre: server umount lustre-MDT0001 complete [ 5002.260571] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 17:28:00 (1773610080) [ 5007.227858] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 5030.883799] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 5045.686516] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5104.164423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5104.507542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5104.547287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5104.642865] Lustre: lustre-MDT0000: new disk, initializing [ 5104.757595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5104.780376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5111.531353] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5127.712083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5127.889467] Lustre: 112112:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5127.955531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5127.960552] Lustre: Skipped 1 previous similar message [ 5128.040913] Lustre: lustre-MDT0001: new disk, initializing [ 5128.151168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5128.295281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5128.309508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5134.432856] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5145.285517] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5151.798965] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5162.059448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5162.632962] Lustre: lustre-OST0000: new disk, initializing [ 5162.648473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5162.792049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5170.823947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5170.845555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5171.109050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5171.963124] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5182.284737] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5188.431922] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5188.751093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5195.198915] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5195.463221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5201.902070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5201.940635] Lustre: Skipped 1 previous similar message [ 5201.961512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5201.964808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5202.003534] Lustre: Skipped 1 previous similar message [ 5204.594976] Lustre: server umount lustre-OST0000 complete [ 5212.137155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5212.147885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5212.172041] Lustre: Skipped 2 previous similar messages [ 5212.184855] Lustre: Skipped 1 previous similar message [ 5217.197905] Lustre: server umount lustre-MDT0000 complete [ 5217.249592] LustreError: 112117:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.291353] LustreError: 112117:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 5222.379137] LustreError: 113184:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5222.414019] LustreError: 113184:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5223.336100] LustreError: 112891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610303 with bad export cookie 2478949963028478683 [ 5223.345339] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5223.381286] LustreError: 112891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5224.205516] Lustre: server umount lustre-MDT0001 complete [ 5237.468536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5238.135782] LustreError: 115929:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5238.280222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5243.362536] LustreError: 115930:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5245.724756] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5248.494870] LustreError: 115929:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5258.685777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5267.070788] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5275.960900] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5282.170311] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5291.757488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5301.361061] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5309.767381] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5316.783396] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 5325.297655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5325.302120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5325.326805] Lustre: Skipped 1 previous similar message [ 5325.329600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5325.362701] LustreError: Skipped 1 previous similar message [ 5327.954206] Lustre: server umount lustre-OST0000 complete [ 5335.530142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5335.546041] Lustre: Skipped 1 previous similar message [ 5339.083376] Lustre: server umount lustre-MDT0000 complete [ 5340.646417] LustreError: 116811:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5340.675391] LustreError: 116811:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5344.385052] LustreError: 115910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610424 with bad export cookie 2478949963028479565 [ 5344.401634] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5344.429534] LustreError: 115910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5345.127924] Lustre: server umount lustre-MDT0001 complete [ 5360.357952] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 5365.060477] Key type lgssc unregistered [ 5365.612834] LNet: 119207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5365.623977] LNetError: 119207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5365.656431] LNet: Removed LNI 192.168.201.120@tcp [ 5366.880395] Key type .llcrypt unregistered [ 5366.883071] Key type ._llcrypt unregistered [ 5391.138164] Key type ._llcrypt registered [ 5391.142253] Key type .llcrypt registered [ 5391.292519] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 5413.363083] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5415.201864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5415.440826] alg: No test for adler32 (adler32-zlib) [ 5416.550487] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 5416.835881] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 5418.544189] Key type lgssc registered [ 5420.576284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5467.369423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5467.409874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5468.728456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5468.762968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5468.876281] Lustre: lustre-MDT0000: new disk, initializing [ 5469.052047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5469.085271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5475.222487] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5492.508938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5492.706353] Lustre: 122331:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5492.891658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5492.906421] Lustre: Skipped 1 previous similar message [ 5493.156712] Lustre: lustre-MDT0001: new disk, initializing [ 5493.325947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5493.400234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5493.424440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5500.570870] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5513.137931] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5519.227411] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5529.241693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5529.686794] Lustre: lustre-OST0000: new disk, initializing [ 5529.690461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5529.818203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.495542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5538.510485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5538.730281] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5539.641236] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5551.535970] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5558.512300] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5558.836451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5565.531398] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5565.902958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5569.506665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5569.507549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.524654] LustreError: Skipped 1 previous similar message [ 5569.525300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5569.544462] Lustre: Skipped 1 previous similar message [ 5569.566082] Lustre: Skipped 1 previous similar message [ 5574.650463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5574.663530] Lustre: Skipped 1 previous similar message [ 5579.748062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5579.753894] Lustre: Skipped 1 previous similar message [ 5583.328141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5583.661705] Lustre: server umount lustre-OST0000 complete [ 5590.004671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5590.007795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5590.034550] Lustre: Skipped 1 previous similar message [ 5590.046308] Lustre: Skipped 1 previous similar message [ 5595.106788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5595.115679] Lustre: Skipped 1 previous similar message [ 5603.300721] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5603.535552] Lustre: server umount lustre-MDT0000 complete [ 5609.563474] LustreError: 122322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610689 with bad export cookie 9070601230156024013 [ 5609.572934] LustreError: 122322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5609.574252] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5610.085709] Lustre: server umount lustre-MDT0001 complete [ 5621.919987] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 17:38:20 (1773610700) [ 5630.393038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.120179] LustreError: 126347:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5631.147735] LustreError: 126347:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5631.216399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.581571] LustreError: 126348:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5637.466805] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5641.708780] LustreError: 126347:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5646.829735] LustreError: 126348:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5649.442736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5649.935845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5657.268878] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5665.943836] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5672.928655] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5680.613323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5680.624717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5680.633861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5680.656566] Lustre: Skipped 1 previous similar message [ 5680.657406] Lustre: Skipped 2 previous similar messages [ 5690.336150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5690.673606] Lustre: server umount lustre-MDT0000 complete [ 5690.861250] LustreError: 127231:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5690.896967] LustreError: 127231:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5695.716826] LustreError: 126327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610776 with bad export cookie 9070601230156024895 [ 5695.730862] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5695.738421] LustreError: 126327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5696.165906] Lustre: server umount lustre-MDT0001 complete [ 5707.301656] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 17:39:46 (1773610786) [ 5718.302272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5734.113188] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5739.232781] LustreError: 128539:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 5765.088488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5775.387930] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5783.888320] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5787.547423] Lustre: server umount lustre-OST0000 complete [ 5800.425428] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 17:41:18 (1773610878) [ 5810.475915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.363747] LustreError: 129873:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.390954] LustreError: 129873:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5811.504761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5819.209756] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5831.156400] LustreError: 129874:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5831.214112] LustreError: 129874:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5834.843830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5842.911685] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5851.862722] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5858.582897] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5870.620099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5871.311688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5871.316346] Lustre: Skipped 1 previous similar message [ 5879.802153] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5888.189536] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5901.794052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5901.806251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5901.823277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5901.830027] Lustre: Skipped 3 previous similar messages [ 5904.158910] Lustre: server umount lustre-MDT0000 complete [ 5907.452571] LustreError: 129869:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5907.487499] LustreError: 129869:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5910.171575] LustreError: 131497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610990 with bad export cookie 9070601230156025560 [ 5910.182690] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5910.187181] LustreError: 131497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5910.792957] Lustre: server umount lustre-MDT0001 complete [ 5926.256966] Lustre: server umount lustre-OST0000 complete [ 5938.599523] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 17:43:37 (1773611017) [ 5948.958068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5949.876386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5956.108696] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5970.978886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5978.272620] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5987.036134] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5992.549575] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6003.449324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6014.040763] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6020.548154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 6023.005314] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6031.716595] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6032.110834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6039.079681] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6039.438311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6046.183267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6046.206977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6046.229157] Lustre: Skipped 2 previous similar messages [ 6046.244984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6046.252110] Lustre: Skipped 3 previous similar messages [ 6048.876686] Lustre: server umount lustre-OST0000 complete [ 6056.419950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6056.439101] Lustre: Skipped 2 previous similar messages [ 6060.631197] Lustre: server umount lustre-MDT0000 complete [ 6061.537585] LustreError: 133193:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.564187] LustreError: 133193:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 6066.114723] LustreError: 133179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611146 with bad export cookie 9070601230156026631 [ 6066.147501] LustreError: 133179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6066.148415] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6066.762990] Lustre: server umount lustre-MDT0001 complete [ 6079.937955] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 17:45:58 (1773611158) [ 6090.325368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6106.213733] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6111.393166] LustreError: 136992:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 6137.188773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6137.199497] Lustre: Skipped 2 previous similar messages [ 6147.662798] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6157.280786] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6168.877405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6169.587806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 6176.538685] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6191.074429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.986319] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6208.511669] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6214.356838] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6221.622759] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6222.172629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6229.086393] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6229.423235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6232.547355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6232.554970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.579920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6232.592361] Lustre: Skipped 2 previous similar messages [ 6239.215481] Lustre: server umount lustre-MDT0000 complete [ 6243.306249] LustreError: 137920:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6243.332261] LustreError: 137920:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 11 previous similar messages [ 6244.842018] LustreError: 136998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611325 with bad export cookie 9070601230156027653 [ 6244.858735] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6244.863431] LustreError: 136998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6245.285870] Lustre: server umount lustre-MDT0001 complete [ 6261.112322] Lustre: server umount lustre-OST0000 complete [ 6274.118834] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 17:49:12 (1773611352) [ 6283.886460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6299.618455] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6304.762665] LustreError: 140825:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 6340.580384] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6349.832075] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6362.743590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6364.447639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6372.757438] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6387.368399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6394.881395] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6404.627134] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6411.534681] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6422.488138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6422.716636] Lustre: lustre-OST0001: new disk, initializing [ 6422.724477] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6422.884212] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6422.894887] Lustre: Skipped 5 previous similar messages [ 6432.920167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6432.940681] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6432.972646] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6433.153239] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6445.175713] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6453.218951] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6453.496842] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6461.028930] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6461.431276] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6465.006764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6465.019821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6465.056296] Lustre: Skipped 2 previous similar messages [ 6465.075580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6465.087219] Lustre: Skipped 6 previous similar messages [ 6470.793680] Lustre: server umount lustre-OST0000 complete [ 6483.458500] Lustre: server umount lustre-OST0001 complete [ 6495.716196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6495.732830] LustreError: Skipped 2 previous similar messages [ 6497.537638] Lustre: server umount lustre-MDT0000 complete [ 6503.178916] LustreError: 140832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611583 with bad export cookie 9070601230156028605 [ 6503.185532] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6503.206144] LustreError: 140832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6504.016901] Lustre: server umount lustre-MDT0001 complete [ 6542.448755] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 17:53:40 (1773611620) [ 6545.316464] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 6547.632435] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 6550.640052] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 17:53:48 (1773611628) [ 6561.345613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6561.884691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6561.929266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6569.767737] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6583.409905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6583.541907] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6583.579649] Lustre: 146792:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6590.614339] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6600.659373] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6607.523349] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6618.994562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6619.328860] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6628.094716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 6629.234495] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6638.253752] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6645.621665] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6645.988339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6653.262798] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6653.704513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6659.052379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6659.071205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6659.078459] LustreError: Skipped 1 previous similar message [ 6659.109482] Lustre: Skipped 6 previous similar messages [ 6671.330830] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6671.706765] Lustre: server umount lustre-OST0000 complete [ 6704.096872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6714.324316] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6723.798358] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6724.398216] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6724.415927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6724.420618] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6724.442075] Lustre: Skipped 1 previous similar message [ 6732.221645] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6732.695896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6739.304460] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6739.559326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6745.500182] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d09050d6000.ost_server_uuid 50 [ 6747.362589] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d09050d6000.ost_server_uuid in FULL state after 0 sec [ 6747.595684] LustreError: 146799:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.20@tcp) returned error from blocking AST (req@00000000866ae892 x1859765400817024 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f03b6f4c/0x7de13fc2beabe417 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.20@tcp remote: 0xc89bb72f43348ff expref: 5 pid: 146799 timeout: 6847 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6747.658940] LustreError: lustre-MDT0000: A client on nid 192.168.201.20@tcp was evicted due to a lock blocking callback time out: rc -107 [ 6747.676972] LustreError: 146789:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.20@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f03b6f4c/0x7de13fc2beabe417 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.20@tcp remote: 0xc89bb72f43348ff expref: 6 pid: 146799 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6753.767739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6753.781955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6753.785500] Lustre: Skipped 15 previous similar messages [ 6758.770176] Lustre: server umount lustre-OST0000 complete [ 6770.609875] Lustre: server umount lustre-MDT0000 complete [ 6770.660017] LustreError: 148242:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6770.706090] LustreError: 148242:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 19 previous similar messages [ 6777.377474] LustreError: 148430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611857 with bad export cookie 9070601230156029788 [ 6777.419660] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6783.930466] Lustre: server umount lustre-MDT0001 complete [ 6800.477298] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6805.068445] Key type lgssc unregistered [ 6805.595970] LNet: 152285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6805.603368] LNetError: 152285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6805.629639] LNet: Removed LNI 192.168.201.120@tcp [ 6807.450429] Key type .llcrypt unregistered [ 6807.458827] Key type ._llcrypt unregistered [ 6839.368232] Key type ._llcrypt registered [ 6839.370069] Key type .llcrypt registered [ 6839.549562] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6841.684207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6841.725643] alg: No test for adler32 (adler32-zlib) [ 6842.919836] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 6843.212198] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 6844.888289] Key type lgssc registered [ 6846.479292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6857.615342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6857.649378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6859.662796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6865.575185] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6868.978712] LustreError: 153689:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6869.011348] LustreError: 153689:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6874.090780] LustreError: 153688:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6878.534741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6879.293492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6885.046511] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6893.182791] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6898.263997] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6907.036690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6907.537589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6915.794264] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6924.280793] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6930.042700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6938.279097] Lustre: Failing over lustre-MDT0000 [ 6938.596787] Lustre: server umount lustre-MDT0000 complete [ 6940.642726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6940.648069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6940.668405] LustreError: 153689:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6940.695680] Lustre: Skipped 1 previous similar message [ 6940.734753] LustreError: 153689:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6945.761373] LustreError: 153688:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6945.787526] LustreError: 153688:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6950.886661] LustreError: 155845:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6950.951336] LustreError: 155845:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6952.008128] LustreError: 154407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.20@tcp arrived at 1773612032 with bad export cookie 11394544370635534482 [ 6952.050024] LustreError: 154407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6956.000389] Lustre: 152997:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773612020/real 1773612020] req@000000004fa311f9 x1859766895801600/t0(0) o400->MGC192.168.201.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773612036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6956.054555] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6961.123519] LustreError: 155535:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6961.187599] LustreError: 155535:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 6963.831462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.176144] LustreError: 156426:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 6966.210150] LustreError: 156426:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000f5f15d7b x1859766895805312/t0(0) o250->MGC192.168.201.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1773612046 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6966.268222] LustreError: 156426:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 6966.328215] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9e218dbbf52871db [ 6966.382640] Lustre: MGC192.168.201.120@tcp: Connection restored to 0@lo (at 0@lo) [ 6966.982789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6967.089546] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6972.399547] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6972.411651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6975.781391] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6979.580630] Lustre: lustre-MDT0000: Denying connection for new client d4630055-68b9-414f-aafd-e0da7587031f (at 192.168.201.20@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 6990.823869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6990.840525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6990.871029] Lustre: Skipped 1 previous similar message [ 6990.890166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6992.877274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6992.892235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.217410] Lustre: server umount lustre-OST0000 complete [ 7000.504847] LustreError: 157106:0:(ldlm_lib.c:2983:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 7000.522056] Lustre: 156459:0:(ldlm_lib.c:2386:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 7000.536852] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 7000.562833] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 7000.585611] LustreError: 156459:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b8f76a66 x1859766895823104/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7000.632418] LustreError: 156459:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7000.645674] LustreError: 156459:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 7000.666441] Lustre: lustre-MDT0000: Recovery over after 0:28, of 2 clients 0 recovered and 2 were evicted. [ 7001.222238] Lustre: server umount lustre-MDT0000 complete [ 7003.109906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7003.150281] LustreError: 153683:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7003.177816] LustreError: 153683:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 7006.169962] LustreError: 153670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612086 with bad export cookie 11394544370635534811 [ 7006.195941] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7006.197227] LustreError: 152994:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 7012.823564] Lustre: server umount lustre-MDT0001 complete [ 7027.968502] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7031.638872] Key type lgssc unregistered [ 7032.076344] LNet: 157844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7032.090795] LNetError: 157844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7032.111195] LNet: Removed LNI 192.168.201.120@tcp [ 7033.187779] Key type .llcrypt unregistered [ 7033.192605] Key type ._llcrypt unregistered [ 7063.524350] Key type ._llcrypt registered [ 7063.526680] Key type .llcrypt registered [ 7063.650467] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7065.379356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7065.452454] alg: No test for adler32 (adler32-zlib) [ 7066.656881] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 7067.105134] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7068.906452] Key type lgssc registered [ 7070.221877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7079.063931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7079.085814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7080.622819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7085.806853] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7090.159378] LustreError: 159245:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7090.191795] LustreError: 159245:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7095.269306] LustreError: 159246:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7099.676258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7100.517613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7107.738319] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7116.594278] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7122.988141] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7132.879139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7133.581635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7141.520185] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7149.522221] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7151.052988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7160.296801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7160.312249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7160.335591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7161.316754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7161.347632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7161.350329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7161.391652] Lustre: Skipped 1 previous similar message [ 7164.832231] Lustre: server umount lustre-OST0000 complete [ 7171.561487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7171.566777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7171.585791] Lustre: Skipped 1 previous similar message [ 7171.607712] Lustre: Skipped 1 previous similar message [ 7176.678276] LustreError: 159246:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7176.779713] Lustre: server umount lustre-MDT0000 complete [ 7181.804746] LustreError: 159242:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7181.814448] LustreError: 159226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612262 with bad export cookie 14607172473934157767 [ 7181.815734] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7181.822782] LustreError: 159242:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7181.885560] LustreError: 159226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7182.511142] Lustre: server umount lustre-MDT0001 complete [ 7198.010236] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7201.823231] Key type lgssc unregistered [ 7202.278695] LNet: 162430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7202.288846] LNetError: 162430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7202.308452] LNet: Removed LNI 192.168.201.120@tcp [ 7203.366086] Key type .llcrypt unregistered [ 7203.368298] Key type ._llcrypt unregistered [ 7242.804278] Key type ._llcrypt registered [ 7242.806793] Key type .llcrypt registered [ 7242.954492] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7244.817171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7244.877319] alg: No test for adler32 (adler32-zlib) [ 7246.195698] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 7246.581794] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7248.328207] Key type lgssc registered [ 7249.931104] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7260.026822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7260.080692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.860065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7268.210935] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7272.419974] LustreError: 164116:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7272.460809] LustreError: 164116:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7277.539833] LustreError: 164117:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7279.308280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.924745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7286.499823] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7294.634723] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7299.838603] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7308.204035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7308.720260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7318.412822] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7326.185109] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7332.238721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 7340.519597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7340.527694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7340.555408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7342.563371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7342.565583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7342.572554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7342.615164] Lustre: Skipped 1 previous similar message [ 7346.561697] Lustre: server umount lustre-OST0000 complete [ 7351.779200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7351.786949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.810213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7357.924682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7357.936952] Lustre: Skipped 3 previous similar messages [ 7358.091796] Lustre: server umount lustre-MDT0000 complete [ 7363.042741] LustreError: 164116:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7363.071061] LustreError: 164116:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7363.249068] LustreError: 164097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612443 with bad export cookie 9379833058841658121 [ 7363.257659] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7363.268622] LustreError: 164097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7363.861741] Lustre: server umount lustre-MDT0001 complete [ 7379.186944] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7382.720906] Key type lgssc unregistered [ 7383.027927] LNet: 167323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7383.033643] LNetError: 167323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7384.105616] LNet: Removed LNI 192.168.201.120@tcp [ 7385.213389] Key type .llcrypt unregistered [ 7385.217634] Key type ._llcrypt unregistered [ 7412.672596] Key type ._llcrypt registered [ 7412.676560] Key type .llcrypt registered [ 7412.827102] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7414.416163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7414.591848] alg: No test for adler32 (adler32-zlib) [ 7415.915898] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 7416.255906] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7418.056206] Key type lgssc registered [ 7419.375210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7432.865812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7432.886974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7434.474895] Lustre: *** cfs_fail_loc=135, val=0*** [ 7434.581343] LustreError: 168831:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000 failed (-2) [ 7434.587609] LustreError: 168831:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.120@tcp: cfg command failed: rc = -2 [ 7434.597154] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7434.597154] [ 7434.608168] LustreError: MGC192.168.201.120@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7434.620795] LustreError: 168792:0:(tgt_mount.c:1820:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 7434.631859] LustreError: 168792:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 7434.639544] LustreError: 168792:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-MDT0000 [ 7434.642970] LustreError: 168792:0:(tgt_mount.c:115:server_deregister_mount()) lustre-MDT0000 not registered [ 7434.747981] Lustre: server umount lustre-MDT0000 complete [ 7434.752212] LustreError: 168792:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7442.481324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7443.303832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7448.078541] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7453.667052] LustreError: 169203:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.685582] LustreError: 169203:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7458.796687] LustreError: 169202:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7459.026964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.620385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7465.628140] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7472.629168] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7477.416875] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7480.290391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7480.301382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7480.327847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7484.387001] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7484.388630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7485.480607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7485.489044] Lustre: Skipped 1 previous similar message [ 7486.690168] Lustre: server umount lustre-MDT0000 complete [ 7490.530906] LustreError: 169913:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7490.545195] LustreError: 169913:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7491.017677] LustreError: 169184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612571 with bad export cookie 17093829285233251158 [ 7491.025079] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7491.031559] LustreError: 169184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7491.582705] Lustre: server umount lustre-MDT0001 complete [ 7506.127527] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7509.425581] Key type lgssc unregistered [ 7509.717757] LNet: 171282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7509.727368] LNetError: 171282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7509.749440] LNet: Removed LNI 192.168.201.120@tcp [ 7510.792770] Key type .llcrypt unregistered [ 7510.799117] Key type ._llcrypt unregistered [ 7539.291549] Key type ._llcrypt registered [ 7539.294361] Key type .llcrypt registered [ 7539.404036] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7539.747518] Key type .llcrypt unregistered [ 7539.750406] Key type ._llcrypt unregistered [ 7560.648887] Key type ._llcrypt registered [ 7560.652352] Key type .llcrypt registered [ 7560.760322] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7561.898992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7562.020547] alg: No test for adler32 (adler32-zlib) [ 7563.287837] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 7563.631783] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7565.360202] Key type lgssc registered [ 7567.059935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7576.763496] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7576.790281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7592.546446] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7597.665785] LustreError: 173383:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 7624.480216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7632.016828] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7638.992599] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7647.914230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7648.796296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7648.889269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 7653.867330] LustreError: 173408:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7653.909156] LustreError: 173408:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7655.153399] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7658.978712] LustreError: 173409:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7659.000931] LustreError: 173409:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7664.110906] LustreError: 174310:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7664.135903] LustreError: 174310:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7667.360623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.127587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7673.187692] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7680.554592] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7685.449751] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7688.882949] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7699.938508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7699.946258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.958857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7704.036789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7704.043632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7704.046162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.378602] Lustre: server umount lustre-OST0000 complete [ 7714.275396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7714.276257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7714.310728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7714.334192] Lustre: Skipped 2 previous similar messages [ 7715.772039] Lustre: server umount lustre-MDT0000 complete [ 7719.393139] LustreError: 174314:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7719.403632] LustreError: 174314:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7719.581590] LustreError: 173389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612799 with bad export cookie 14215721498830042306 [ 7719.586137] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7719.588050] LustreError: 173389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7719.860820] Lustre: server umount lustre-MDT0001 complete [ 7732.077428] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7735.294329] Key type lgssc unregistered [ 7735.624955] LNet: 176954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7735.635136] LNetError: 176954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7735.649849] LNet: Removed LNI 192.168.201.120@tcp [ 7736.585658] Key type .llcrypt unregistered [ 7736.587599] Key type ._llcrypt unregistered [ 7764.909797] Key type ._llcrypt registered [ 7764.912346] Key type .llcrypt registered [ 7765.018570] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7766.620649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7766.665795] alg: No test for adler32 (adler32-zlib) [ 7767.755200] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 7768.152792] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7769.920275] Key type lgssc registered [ 7771.350040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7779.673359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7779.716216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7781.221302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7785.864311] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7791.593537] LustreError: 178351:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7791.624326] LustreError: 178351:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7795.929564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7796.311753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7801.422764] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7807.394399] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7811.686257] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7819.582747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7820.073852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7826.230451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 7826.433907] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7833.020930] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7838.691421] Lustre: Failing over lustre-MDT0000 [ 7839.064925] Lustre: server umount lustre-MDT0000 complete [ 7840.884331] LustreError: 179971:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7841.766723] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7841.796042] Lustre: Skipped 2 previous similar messages [ 7845.967335] LustreError: 178346:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7846.001852] LustreError: 178346:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 7851.094624] LustreError: 178347:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7851.111673] LustreError: 178347:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 7856.211221] LustreError: 178346:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7856.241530] LustreError: 178346:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 7858.151355] Lustre: 177662:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773612922/real 1773612922] req@000000002be81fba x1859767865730304/t0(0) o400->MGC192.168.201.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773612938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7858.184231] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7858.422452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7867.469608] LustreError: 178345:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7867.485206] LustreError: 178345:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 12 previous similar messages [ 7868.389103] LustreError: 177660:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bc1348ec x1859767865735552/t0(0) o250->MGC192.168.201.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7868.739449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7868.805440] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7872.596419] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7873.200392] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7874.037313] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7874.071305] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 7874.152567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 7878.433620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 7884.521308] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 7892.822877] Lustre: server umount lustre-OST0000 complete [ 7899.619643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7899.622059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7899.632468] Lustre: Skipped 1 previous similar message [ 7899.642963] Lustre: Skipped 1 previous similar message [ 7904.738313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7905.825885] Lustre: server umount lustre-MDT0000 complete [ 7908.834325] Lustre: 177661:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773612973/real 1773612973] req@00000000b578e8c9 x1859767865761152/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773612989 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7908.856745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7908.873019] LustreError: 179462:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7908.895743] LustreError: 179462:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 7908.994220] LustreError: 178331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612989 with bad export cookie 17517725840316507871 [ 7909.002680] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7909.012634] LustreError: 178331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7909.293957] Lustre: server umount lustre-MDT0001 complete [ 7920.393590] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7924.417272] Key type lgssc unregistered [ 7924.795696] LNet: 182829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7924.801277] LNetError: 182829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7924.818260] LNet: Removed LNI 192.168.201.120@tcp [ 7925.971523] Key type .llcrypt unregistered [ 7925.978370] Key type ._llcrypt unregistered [ 7949.490992] Key type ._llcrypt registered [ 7949.496653] Key type .llcrypt registered [ 7949.678176] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7950.840320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7950.886677] alg: No test for adler32 (adler32-zlib) [ 7951.945918] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 7952.171716] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7953.824969] Key type lgssc registered [ 7954.768671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7962.430463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7962.444091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7964.140802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7969.045689] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7974.372114] LustreError: 184227:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7974.392718] LustreError: 184227:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7978.587622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7978.925256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7983.662369] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7989.318687] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7993.798611] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8001.410586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8001.775682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.784162] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8008.954647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 8014.144860] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8018.109252] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 8023.662881] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 8035.002733] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 8050.146435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8050.148702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8050.157218] LustreError: Skipped 1 previous similar message [ 8050.159488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8050.177707] Lustre: Skipped 1 previous similar message [ 8053.729308] Lustre: server umount lustre-OST0000 complete [ 8060.397350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8060.405071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8060.423716] Lustre: Skipped 1 previous similar message [ 8060.434694] Lustre: Skipped 2 previous similar messages [ 8063.958635] Lustre: server umount lustre-MDT0000 complete [ 8065.506220] LustreError: 184222:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8065.521807] LustreError: 184222:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8068.384325] LustreError: 184209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773613148 with bad export cookie 6101228311745931485 [ 8068.387468] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8068.419878] LustreError: 184209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8068.777742] Lustre: server umount lustre-MDT0001 complete [ 8082.194498] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8085.860874] Key type lgssc unregistered [ 8086.303101] LNet: 187616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8086.313881] LNetError: 187616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8086.335740] LNet: Removed LNI 192.168.201.120@tcp [ 8087.439579] Key type .llcrypt unregistered [ 8087.447443] Key type ._llcrypt unregistered [ 8111.226480] Key type ._llcrypt registered [ 8111.228554] Key type .llcrypt registered [ 8111.338358] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8112.411495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8112.623577] alg: No test for adler32 (adler32-zlib) [ 8113.699950] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 8113.943718] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8115.648782] Key type lgssc registered [ 8116.967996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8126.652463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8126.681806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8128.312221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8133.776541] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8137.704349] LustreError: 189013:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8137.726740] LustreError: 189013:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 8142.832626] LustreError: 189014:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8143.958841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8144.399839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8149.610352] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8156.218285] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8160.107771] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8168.060357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8168.471418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8173.581465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 8175.509643] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8181.631945] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8188.326359] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 8203.062662] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 8213.350189] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 8222.219263] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 8234.982377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8234.999046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8235.012680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8235.021154] Lustre: Skipped 1 previous similar message [ 8235.032889] Lustre: Skipped 1 previous similar message [ 8240.104394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8240.111034] Lustre: Skipped 1 previous similar message [ 8240.417311] Lustre: server umount lustre-OST0000 complete [ 8245.226419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8245.230630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8245.243838] Lustre: Skipped 1 previous similar message [ 8245.258874] Lustre: Skipped 1 previous similar message [ 8250.338644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8250.344465] Lustre: Skipped 1 previous similar message [ 8250.921869] Lustre: server umount lustre-MDT0000 complete [ 8254.822073] LustreError: 188995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773613335 with bad export cookie 13017873635007358148 [ 8254.823972] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8254.836043] LustreError: 188995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8255.148703] Lustre: server umount lustre-MDT0001 complete [ 8265.903488] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8268.796357] Key type lgssc unregistered [ 8269.103718] LNet: 193825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8269.110856] LNetError: 193825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8269.127076] LNet: Removed LNI 192.168.201.120@tcp [ 8269.933266] Key type .llcrypt unregistered [ 8269.937602] Key type ._llcrypt unregistered [ 8305.941553] Key type ._llcrypt registered [ 8305.944363] Key type .llcrypt registered [ 8306.050495] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8308.288160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8308.420820] alg: No test for adler32 (adler32-zlib) [ 8310.651766] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 8311.561436] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8313.880260] Key type lgssc registered [ 8317.954333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8334.949584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8334.987188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8337.183411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8345.309206] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8347.635536] LustreError: 195224:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8347.672622] LustreError: 195224:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 8352.739894] LustreError: 195225:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8357.867219] LustreError: 195224:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8361.989424] LustreError: 195225:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8362.620915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8363.597662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8370.141185] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8379.056939] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8385.853221] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8397.843982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8398.621349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8406.886038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 8407.963541] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8418.432819] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8442.850704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8442.864342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8442.898173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8443.880064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8443.891960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8443.907722] Lustre: Skipped 1 previous similar message [ 8447.578328] Lustre: server umount lustre-OST0000 complete [ 8454.125714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8454.138973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8454.151795] Lustre: Skipped 1 previous similar message [ 8459.233804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8459.248386] Lustre: Skipped 2 previous similar messages [ 8459.827086] Lustre: server umount lustre-MDT0000 complete [ 8464.359693] LustreError: 195225:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8464.394186] LustreError: 195225:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8464.930504] LustreError: 195206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773613545 with bad export cookie 736431385206622428 [ 8464.931602] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8464.947720] LustreError: 195206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8465.395821] Lustre: server umount lustre-MDT0001 complete [ 8482.709672] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8487.868733] Key type lgssc unregistered [ 8488.439495] LNet: 198566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8488.451502] LNetError: 198566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8488.472755] LNet: Removed LNI 192.168.201.120@tcp [ 8490.546843] Key type .llcrypt unregistered [ 8490.548641] Key type ._llcrypt unregistered [ 8521.416492] Key type ._llcrypt registered [ 8521.422727] Key type .llcrypt registered [ 8521.659915] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8523.641285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8523.738657] alg: No test for adler32 (adler32-zlib) [ 8525.103736] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 8525.514771] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8527.323842] Key type lgssc registered [ 8529.201622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8539.915821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8539.950854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8541.970884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8549.504836] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8552.442199] LustreError: 199820:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8552.511862] LustreError: 199820:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 8557.544943] LustreError: 199821:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8562.659907] LustreError: 199820:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8562.994907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8563.765448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8570.235392] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8579.528381] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8585.423283] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8599.220855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8599.953909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8610.700382] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8612.257711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 8620.023249] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8643.048313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8643.057463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8643.081777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8648.168042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8648.176182] Lustre: Skipped 1 previous similar message [ 8649.569786] Lustre: server umount lustre-OST0000 complete [ 8655.844590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8655.863435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8655.891883] Lustre: Skipped 1 previous similar message [ 8655.904061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8658.403912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8658.405563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8658.481044] Lustre: Skipped 1 previous similar message [ 8661.956494] Lustre: server umount lustre-MDT0000 complete [ 8663.526184] LustreError: 199820:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8663.558993] LustreError: 199820:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8668.650359] LustreError: 200933:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8668.702405] LustreError: 200933:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8669.073494] LustreError: 199800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773613749 with bad export cookie 8044320811045526271 [ 8669.092648] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8669.112712] LustreError: 199800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8669.969495] Lustre: server umount lustre-MDT0001 complete [ 8692.277696] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8697.089731] Key type lgssc unregistered [ 8697.482359] LNet: 203179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8697.507472] LNetError: 203179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8697.549288] LNet: Removed LNI 192.168.201.120@tcp [ 8698.859411] Key type .llcrypt unregistered [ 8698.864991] Key type ._llcrypt unregistered [ 8729.214406] Key type ._llcrypt registered [ 8729.216977] Key type .llcrypt registered [ 8729.437927] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8732.071586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8732.130253] alg: No test for adler32 (adler32-zlib) [ 8733.643578] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 8734.305158] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8736.251326] Key type lgssc registered [ 8738.577424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8754.026854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8754.084907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8756.243353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8763.733421] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8766.443104] LustreError: 204432:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8766.508878] LustreError: 204432:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 8771.555476] LustreError: 204433:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.704532] LustreError: 204432:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8777.205773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8778.052740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8784.686681] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8795.193349] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8802.851058] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8814.000825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8814.692751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8824.415536] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8826.004362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 8835.414824] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8850.162836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8850.518517] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8851.160279] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8861.692780] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8872.197626] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8874.925782] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8888.512951] Lustre: Permanently deactivating lustre-OST0001 [ 8888.552403] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 8895.182675] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 8905.734661] Lustre: Permanently reactivating lustre-OST0001 [ 8905.762561] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 8905.771248] Lustre: Skipped 2 previous similar messages [ 8914.638722] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8914.678183] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 8914.710225] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8914.729954] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8914.851693] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8919.020546] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8919.072849] Lustre: Skipped 2 previous similar messages [ 8919.083783] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8919.111868] Lustre: Skipped 1 previous similar message [ 8924.131812] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8924.154997] Lustre: Skipped 1 previous similar message [ 8929.286210] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8929.312392] Lustre: Skipped 1 previous similar message [ 8931.296148] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8931.415583] Lustre: server umount lustre-OST0001 complete [ 8934.387615] LustreError: 206323:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8934.418779] LustreError: 206323:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8938.977848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8938.989297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8939.026226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8939.491814] LustreError: 206029:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8939.567974] LustreError: 206029:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8944.628575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8944.652822] Lustre: Skipped 3 previous similar messages [ 8944.974926] Lustre: server umount lustre-OST0000 complete [ 8952.294230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8952.313705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8952.350091] Lustre: Skipped 1 previous similar message [ 8954.865831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8954.879963] Lustre: Skipped 2 previous similar messages [ 8958.253850] Lustre: server umount lustre-MDT0000 complete [ 8959.980531] LustreError: 205144:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8960.009684] LustreError: 205144:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 8964.935040] LustreError: 204413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614045 with bad export cookie 7829301748763210841 [ 8964.945304] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8964.951895] LustreError: 204413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8965.858785] Lustre: server umount lustre-MDT0001 complete [ 8987.167302] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8992.768791] Key type lgssc unregistered [ 8993.234808] LNet: 209048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8993.259335] LNetError: 209048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8993.297752] LNet: Removed LNI 192.168.201.120@tcp [ 8995.239857] Key type .llcrypt unregistered [ 8995.242892] Key type ._llcrypt unregistered [ 9068.637587] Key type ._llcrypt registered [ 9068.647848] Key type .llcrypt registered [ 9068.852583] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9071.246393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9071.486923] alg: No test for adler32 (adler32-zlib) [ 9073.266251] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 9073.663654] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 9075.529457] Key type lgssc registered [ 9077.334497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9091.344125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9091.403959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.064781] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9093.090361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9093.496715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9101.498444] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9116.581179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9116.940130] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9117.007427] Lustre: 211242:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9117.472527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9124.629736] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9134.104518] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9140.588939] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9151.847157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9152.311489] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9152.881419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9161.527877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 9162.980110] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9172.650270] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9177.049285] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 9183.364483] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 9188.709050] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 9197.689763] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9206.463608] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 9231.930446] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 9231.943489] Lustre: Skipped 2 previous similar messages [ 9266.993923] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9267.007864] Lustre: Skipped 3 previous similar messages [ 9334.480858] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9334.498181] Lustre: Skipped 7 previous similar messages [ 9366.505809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9366.514133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9366.517309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9366.559400] Lustre: Skipped 1 previous similar message [ 9371.629172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9371.639431] Lustre: Skipped 2 previous similar messages [ 9372.184407] Lustre: server umount lustre-OST0000 complete [ 9378.788164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9378.808414] LustreError: Skipped 1 previous similar message [ 9378.830326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9378.901567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9381.861495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9381.881046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9385.173464] Lustre: server umount lustre-MDT0000 complete [ 9391.192643] LustreError: 211236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614471 with bad export cookie 740434078864143247 [ 9391.196905] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9391.206264] LustreError: 211236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9391.708632] Lustre: server umount lustre-MDT0001 complete [ 9409.305958] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 9416.097580] Key type lgssc unregistered [ 9416.611310] LNet: 215701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9416.618329] LNetError: 215701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9417.648211] LNet: Removed LNI 192.168.201.120@tcp [ 9419.252308] Key type .llcrypt unregistered [ 9419.255993] Key type ._llcrypt unregistered [ 9464.750549] Key type ._llcrypt registered [ 9464.756674] Key type .llcrypt registered [ 9464.987201] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9468.555415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9468.583679] alg: No test for adler32 (adler32-zlib) [ 9469.808760] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 9470.165958] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 9471.968486] Key type lgssc registered [ 9473.963323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9487.518765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9487.558409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9489.372673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9497.173391] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9499.621734] LustreError: 217105:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9499.664027] LustreError: 217105:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 9504.743242] LustreError: 217104:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9509.868258] LustreError: 217105:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9511.078535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.729626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9519.235757] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9529.727504] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9535.751418] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9546.135714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9546.787111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9558.429462] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9559.065416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 9568.103842] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9594.853761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9594.868857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9594.881530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9594.914481] Lustre: Skipped 1 previous similar message [ 9594.946414] Lustre: Skipped 1 previous similar message [ 9597.220528] Lustre: server umount lustre-OST0000 complete [ 9604.065164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9604.069550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9604.082159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9605.092381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9605.095181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9609.112093] Lustre: server umount lustre-MDT0000 complete [ 9610.211164] LustreError: 217105:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9610.251269] LustreError: 217105:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 9615.309302] LustreError: 217083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614695 with bad export cookie 13117991658672264554 [ 9615.327402] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9615.331628] LustreError: 217083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9615.827391] Lustre: server umount lustre-MDT0001 complete [ 9633.488551] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 9637.933578] Key type lgssc unregistered [ 9638.459670] LNet: 220468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9638.466917] LNetError: 220468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9639.528265] LNet: Removed LNI 192.168.201.120@tcp [ 9641.060653] Key type .llcrypt unregistered [ 9641.066892] Key type ._llcrypt unregistered [ 9679.331952] Key type ._llcrypt registered [ 9679.334959] Key type .llcrypt registered [ 9679.576661] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9681.542670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9681.624541] alg: No test for adler32 (adler32-zlib) [ 9683.234298] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 9683.742183] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 9685.577214] Key type lgssc registered [ 9688.244799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9965.081319] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 9970.228326] Key type lgssc unregistered [ 9970.667561] LNet: 222463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9970.672689] LNetError: 222463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9970.704036] LNet: Removed LNI 192.168.201.120@tcp [ 9972.244509] Key type .llcrypt unregistered [ 9972.247277] Key type ._llcrypt unregistered [10017.291960] Key type ._llcrypt registered [10017.302795] Key type .llcrypt registered [10017.635961] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [10045.544790] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10049.092507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10049.166081] alg: No test for adler32 (adler32-zlib) [10050.760140] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [10051.265577] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [10053.141156] Key type lgssc registered [10055.237265] Lustre: Echo OBD driver; http://www.lustre.org/ [10110.967094] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9672 sec ========== 18:53:08 (1773615188) [10113.450341] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:53:11 (1773615191) === [10119.238970] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:53:17 (1773615197) === [10181.431716] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [10185.028063] Key type lgssc unregistered [10185.283713] LNet: 227660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10185.294794] LNetError: 227660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10185.341279] LNet: Removed LNI 192.168.201.120@tcp [10186.265618] Key type .llcrypt unregistered [10186.269284] Key type ._llcrypt unregistered