[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 502674824 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003312] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.007866] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009022] pid_max: default: 32768 minimum: 301 [ 0.010183] LSM: Security Framework initializing [ 0.011085] Yama: becoming mindful. [ 0.012051] SELinux: Initializing. [ 0.013090] *** VALIDATE selinux *** [ 0.021827] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026627] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027193] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031148] *** VALIDATE tmpfs *** [ 0.033087] *** VALIDATE proc *** [ 0.034358] *** VALIDATE cgroup *** [ 0.035018] *** VALIDATE cgroup2 *** [ 0.037028] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038197] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040036] Spectre V2 : User space: Vulnerable [ 0.041016] Speculative Store Bypass: Vulnerable [ 0.044143] debug: unmapping init [mem 0xffffffffb8e59000-0xffffffffb8e60fff] [ 0.047000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047848] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048033] ... version: 2 [ 0.049018] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051018] ... value mask: 0000ffffffffffff [ 0.052019] ... max period: 00007fffffffffff [ 0.053036] ... fixed-purpose events: 3 [ 0.054020] ... event mask: 000000070000000f [ 0.055381] rcu: Hierarchical SRCU implementation. [ 0.057816] smp: Bringing up secondary CPUs ... [ 0.058753] x86: Booting SMP configuration: [ 0.059041] .... node #0, CPUs: #1 #2 #3 [ 0.062503] smp: Brought up 1 node, 4 CPUs [ 0.064020] smpboot: Max logical packages: 1 [ 0.065028] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147357] node 0 deferred pages initialised in 80ms [ 0.151134] devtmpfs: initialized [ 0.153353] x86/mm: Memory block size: 128MB [ 0.157522] gcov: version magic: 0x41383552 [ 0.160456] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.164095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.167448] pinctrl core: initialized pinctrl subsystem [ 0.169305] [ 0.169696] ************************************************************* [ 0.172016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175016] ** ** [ 0.177012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.180020] ** ** [ 0.182010] ** This means that this kernel is built to expose internal ** [ 0.183011] ** IOMMU data structures, which may compromise security on ** [ 0.185016] ** your system. ** [ 0.187013] ** ** [ 0.189012] ** If you see this message and you are not debugging the ** [ 0.190011] ** kernel, report this immediately to your vendor! ** [ 0.192029] ** ** [ 0.195033] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197012] ************************************************************* [ 0.199986] NET: Registered protocol family 16 [ 0.202527] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.205099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211136] cpuidle: using governor menu [ 0.213000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215815] PCI: Using configuration type 1 for base access [ 0.218146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228177] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.231073] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.235095] cryptd: max_cpu_qlen set to 1000 [ 0.237292] ACPI: Added _OSI(Module Device) [ 0.239137] ACPI: Added _OSI(Processor Device) [ 0.241015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.243019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248442] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256412] ACPI: Interpreter enabled [ 0.258138] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260047] ACPI: Using IOAPIC for interrupt routing [ 0.261134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.265417] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.274987] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.277053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.279025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282118] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.286764] acpiphp: Slot [2] registered [ 0.288230] acpiphp: Slot [5] registered [ 0.289094] acpiphp: Slot [6] registered [ 0.290248] acpiphp: Slot [7] registered [ 0.292215] acpiphp: Slot [8] registered [ 0.294192] acpiphp: Slot [9] registered [ 0.295288] acpiphp: Slot [10] registered [ 0.297200] acpiphp: Slot [3] registered [ 0.299155] acpiphp: Slot [4] registered [ 0.301154] acpiphp: Slot [11] registered [ 0.302138] acpiphp: Slot [12] registered [ 0.304176] acpiphp: Slot [13] registered [ 0.306155] acpiphp: Slot [14] registered [ 0.308137] acpiphp: Slot [15] registered [ 0.310146] acpiphp: Slot [16] registered [ 0.311131] acpiphp: Slot [17] registered [ 0.313147] acpiphp: Slot [18] registered [ 0.315248] acpiphp: Slot [19] registered [ 0.316136] acpiphp: Slot [20] registered [ 0.318150] acpiphp: Slot [21] registered [ 0.320141] acpiphp: Slot [22] registered [ 0.321105] acpiphp: Slot [23] registered [ 0.322110] acpiphp: Slot [24] registered [ 0.324150] acpiphp: Slot [25] registered [ 0.325116] acpiphp: Slot [26] registered [ 0.326200] acpiphp: Slot [27] registered [ 0.327138] acpiphp: Slot [28] registered [ 0.329101] acpiphp: Slot [29] registered [ 0.330103] acpiphp: Slot [30] registered [ 0.331109] acpiphp: Slot [31] registered [ 0.332110] PCI host bridge to bus 0000:00 [ 0.334029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.341027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348269] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351279] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.361810] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.367063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.372028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.374024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378782] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380921] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384067] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.388134] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.419000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.437022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.444017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.471702] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.485023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.493022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.506022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.515088] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.520019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.524016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.537021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.545153] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.555024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.560020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.589024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.611160] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.616016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.621017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.633017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.640839] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.645015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.650018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.661018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.670582] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.672310] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.675383] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.677374] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.678193] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.683204] iommu: Default domain type: Passthrough [ 0.685546] SCSI subsystem initialized [ 0.686152] ACPI: bus type USB registered [ 0.687136] usbcore: registered new interface driver usbfs [ 0.688086] usbcore: registered new interface driver hub [ 0.690089] usbcore: registered new device driver usb [ 0.691193] pps_core: LinuxPPS API ver. 1 registered [ 0.692012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.695101] PTP clock support registered [ 0.697142] EDAC MC: Ver: 3.0.0 [ 0.699150] PCI: Using ACPI for IRQ routing [ 0.700992] NetLabel: Initializing [ 0.702012] NetLabel: domain hash size = 128 [ 0.703009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.704152] NetLabel: unlabeled traffic allowed by default [ 0.706186] vgaarb: loaded [ 0.707301] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.709016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.715666] clocksource: Switched to clocksource kvm-clock [ 0.825725] VFS: Disk quotas dquot_6.6.0 [ 0.827382] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.829885] *** VALIDATE ramfs *** [ 0.830931] *** VALIDATE hugetlbfs *** [ 0.832914] pnp: PnP ACPI init [ 0.835600] pnp: PnP ACPI: found 6 devices [ 0.855193] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.858073] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.860783] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.862298] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.863867] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.865548] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.867588] NET: Registered protocol family 2 [ 0.869609] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.873631] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.876302] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.882557] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.885350] TCP: Hash tables configured (established 65536 bind 65536) [ 0.887852] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.890389] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.892867] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.895696] NET: Registered protocol family 1 [ 0.898292] RPC: Registered named UNIX socket transport module. [ 0.900700] RPC: Registered udp transport module. [ 0.902439] RPC: Registered tcp transport module. [ 0.904193] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.906182] NET: Registered protocol family 44 [ 0.907474] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.909213] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.911081] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.913048] PCI: CLS 0 bytes, default 64 [ 0.914485] Unpacking initramfs... [ 2.301709] debug: unmapping init [mem 0xffff9571fcc54000-0xffff9571fffbffff] [ 2.305042] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.306297] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.307915] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.849693] Initialise system trusted keyrings [ 2.851438] Key type blacklist registered [ 2.853777] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.863661] zbud: loaded [ 2.866873] *** VALIDATE nfs *** [ 2.867767] *** VALIDATE nfs4 *** [ 2.868966] pstore: using deflate compression [ 2.871959] Platform Keyring initialized [ 2.989487] NET: Registered protocol family 38 [ 2.991394] Key type asymmetric registered [ 2.992938] Asymmetric key parser 'x509' registered [ 2.995107] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.998338] io scheduler mq-deadline registered [ 3.000168] io scheduler kyber registered [ 3.001874] io scheduler bfq registered [ 3.004089] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.007257] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.010120] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.012906] ACPI: Power Button [PWRF] [ 3.018429] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.024799] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.039296] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.046206] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.061793] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.089378] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.119633] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.127559] Non-volatile memory driver v1.3 [ 3.129205] Linux agpgart interface v0.103 [ 3.160640] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.166540] vda: detected capacity change from 0 to 74690560 [ 3.181191] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.184279] vdb: detected capacity change from 0 to 1073741824 [ 3.202258] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.205697] vdc: detected capacity change from 0 to 2621440000 [ 3.218618] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.221986] vdd: detected capacity change from 0 to 2621440000 [ 3.238308] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.241425] vde: detected capacity change from 0 to 4294967296 [ 3.264579] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.267543] vdf: detected capacity change from 0 to 4294967296 [ 3.278176] libphy: Fixed MDIO Bus: probed [ 3.293467] usbcore: registered new interface driver usbserial_generic [ 3.295497] usbserial: USB Serial support registered for generic [ 3.298172] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.301591] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.302985] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.305833] mousedev: PS/2 mouse device common for all mice [ 3.308655] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.310415] rtc_cmos 00:05: RTC can wake from S4 [ 3.315097] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.316420] rtc_cmos 00:05: registered as rtc0 [ 3.320434] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.322430] intel_pstate: CPU model not supported [ 3.324241] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.327050] hid: raw HID events driver (C) Jiri Kosina [ 3.329589] usbcore: registered new interface driver usbhid [ 3.331279] usbhid: USB HID core driver [ 3.332514] drop_monitor: Initializing network drop monitor service [ 3.334248] Initializing XFRM netlink socket [ 3.336125] NET: Registered protocol family 10 [ 3.338704] Segment Routing with IPv6 [ 3.339732] NET: Registered protocol family 17 [ 3.342151] mpls_gso: MPLS GSO support [ 3.347907] RAS: Correctable Errors collector initialized. [ 3.350550] AVX version of gcm_enc/dec engaged. [ 3.352423] AES CTR mode by8 optimization enabled [ 3.428392] sched_clock: Marking stable (3428357579, 0)->(4408497312, -980139733) [ 3.432154] registered taskstats version 1 [ 3.434224] Loading compiled-in X.509 certificates [ 3.436539] zswap: loaded using pool lzo/zbud [ 3.459927] Key type big_key registered [ 3.471520] Key type encrypted registered [ 3.473081] ima: No TPM chip found, activating TPM-bypass! [ 3.474928] ima: Allocated hash algorithm: sha1 [ 3.476549] ima: No architecture policies found [ 3.478056] evm: Initialising EVM extended attributes: [ 3.479780] evm: security.selinux [ 3.481257] evm: security.ima [ 3.482178] evm: security.capability [ 3.483288] evm: HMAC attrs: 0x1 [ 3.485280] rtc_cmos 00:05: setting system clock to 2026-07-25 00:04:27 UTC (1784937867) [ 3.489506] debug: unmapping init [mem 0xffffffffb9e03000-0xffffffffb9ffffff] [ 3.491633] debug: unmapping init [mem 0xffffffffb8b82000-0xffffffffb8e58fff] [ 3.499051] Write protecting the kernel read-only data: 28672k [ 3.501278] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 3.503185] debug: unmapping init [mem 0xffffffffb7b14000-0xffffffffb7bfffff] [ 3.533101] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.541051] systemd[1]: Detected virtualization kvm. [ 3.542747] systemd[1]: Detected architecture x86-64. [ 3.544744] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.572930] systemd[1]: No hostname configured. [ 3.574701] systemd[1]: Set hostname to . [ 3.576741] random: systemd: uninitialized urandom read (16 bytes read) [ 3.578693] systemd[1]: Initializing machine ID from random generator. [ 3.620474] random: ln: uninitialized urandom read (6 bytes read) [ 3.698309] random: systemd: uninitialized urandom read (16 bytes read) [ 3.702252] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.706891] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.711524] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.301595] device-mapper: uevent: version 1.0.3 [ 4.303396] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.958733] virtio_net virtio0 ens2: renamed from eth0 [ 4.973276] random: fast init done [ 5.006070] scsi host0: ata_piix [ 5.031562] scsi host1: ata_piix [ 5.032868] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.034586] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.333073] dracut-initqueue[591]: RTNETLINK answers: File exists [ 9.990121] random: crng init done [ 9.991484] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.274325] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.401579] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.673459] SELinux: Disabled at runtime. [ 11.733319] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.739247] systemd[1]: Detected virtualization kvm. [ 11.740590] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.186879] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.190148] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.194427] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.198655] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.201112] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.209353] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.229871] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ 12.315967] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.615429] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.912977] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.936624] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.044673] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.061779] EDAC sbridge: Ver: 1.1.2 [ 14.401493] Key type dns_resolver registered [ 14.724674] NFS: Registering the id_resolver key type [ 14.726322] Key type id_resolver registered [ 14.727639] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg109-server login: [ 40.565614] libcfs: loading out-of-tree module taints kernel. [ 40.591147] Key type ._llcrypt registered [ 40.592908] Key type .llcrypt registered [ 40.644708] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 54.174925] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 55.812592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 55.822346] alg: No test for adler32 (adler32-zlib) [ 57.355467] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 58.297473] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 60.048373] Key type lgssc registered [ 61.660826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.043863] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 119.595325] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 132.911749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 132.958902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.263328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 134.287461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 134.373156] Lustre: lustre-MDT0000: new disk, initializing [ 134.444516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.463505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 139.521815] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.368569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.494336] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 154.526708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 154.533069] Lustre: Skipped 1 previous similar message [ 154.594221] Lustre: lustre-MDT0001: new disk, initializing [ 154.645543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 154.669294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 154.682249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 159.589426] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.086749] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 175.878773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 176.218835] Lustre: lustre-OST0000: new disk, initializing [ 176.222508] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 176.232793] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 176.386361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.348407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 182.374147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 182.518412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 184.836292] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.502015] hrtimer: interrupt took 12413781 ns [ 202.332273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.525589] Lustre: lustre-OST0001: new disk, initializing [ 202.528510] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 202.535792] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.710625] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.979246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 209.002779] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 209.171757] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 211.931546] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.378806] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 236.809031] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 243.770607] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing check_logdir /tmp/testlogs/ [ 249.227180] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing yml_node [ 254.304842] Lustre: DEBUG MARKER: Client: 2.17.54.203 [ 256.727216] Lustre: DEBUG MARKER: MDS: 2.17.54.203 [ 259.797032] Lustre: DEBUG MARKER: OSS: 2.17.54.203 [ 261.631420] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 24 20:08:44 EDT 2026 [ 282.301881] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 284.263596] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.543205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 301.546796] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 301.565653] Lustre: Skipped 3 previous similar messages [ 301.585806] Lustre: Skipped 3 previous similar messages [ 306.668145] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 306.680673] Lustre: Skipped 3 previous similar messages [ 307.010365] Lustre: server umount lustre-MDT0000 complete [ 310.451581] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938174 with bad export cookie 2957757382624126047 [ 310.454848] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 310.461353] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 311.779879] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 311.800720] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.109@tcp (stopping) [ 311.809184] Lustre: Skipped 1 previous similar message [ 315.816898] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.109@tcp (stopping) [ 315.833331] Lustre: Skipped 1 previous similar message [ 316.901220] Lustre: server umount lustre-MDT0001 complete [ 320.989757] Lustre: server umount lustre-OST0000 complete [ 324.662751] Lustre: server umount lustre-OST0001 complete [ 332.384315] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 342.076850] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 382.524713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 382.781589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 382.817052] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 382.946280] Lustre: lustre-MDT0000: new disk, initializing [ 383.060820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 383.080878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.853317] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 401.005722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.163863] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 401.276333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 401.283469] Lustre: Skipped 1 previous similar message [ 401.358513] Lustre: lustre-MDT0001: new disk, initializing [ 401.469150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 401.536601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 401.570402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 406.821676] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 417.644825] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 424.687559] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 433.354452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 433.783949] Lustre: lustre-OST0000: new disk, initializing [ 433.790287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 433.807120] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 433.951832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.757462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 435.779844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 435.919798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 441.931790] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 454.646096] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 460.811853] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 461.012805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 466.650367] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 466.851531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 472.033670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 472.040511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 472.045382] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 472.048475] Lustre: Skipped 2 previous similar messages [ 472.069892] Lustre: Skipped 1 previous similar message [ 475.515634] Lustre: server umount lustre-OST0000 complete [ 482.276035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 482.290133] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 482.297426] Lustre: Skipped 1 previous similar message [ 482.309348] Lustre: Skipped 1 previous similar message [ 485.891949] Lustre: server umount lustre-MDT0000 complete [ 487.394325] LustreError: 15080:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.417150] LustreError: 15080:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 6 previous similar messages [ 489.528104] LustreError: 17043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938353 with bad export cookie 2957757382624127580 [ 489.536349] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.953893] Lustre: server umount lustre-MDT0001 complete [ 493.912447] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 20:12:36 (1784938356) [ 501.432550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.035458] LustreError: 19309:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.108076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.789351] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 507.392882] LustreError: 19310:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.485170] LustreError: 19309:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.988839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.056408] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 531.382815] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 537.976288] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 545.293365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 551.632890] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 558.683718] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 570.867203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 192.168.201.109@tcp failed: rc = -107 [ 570.887837] LustreError: Skipped 1 previous similar message [ 570.896236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 570.925025] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 572.391088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 576.139894] Lustre: server umount lustre-OST0000 complete [ 582.631946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 582.657598] Lustre: Skipped 2 previous similar messages [ 587.322280] Lustre: server umount lustre-MDT0000 complete [ 587.745213] LustreError: 19310:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.774825] LustreError: 19310:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 591.936496] LustreError: 19289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938455 with bad export cookie 2957757382624128455 [ 591.952305] LustreError: 19289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 591.953848] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 592.499413] Lustre: server umount lustre-MDT0001 complete [ 608.828366] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 614.064189] Key type lgssc unregistered [ 614.460752] LNet: 22845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.465146] LNetError: 22845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.485827] LNet: Removed LNI 192.168.201.109@tcp [ 616.036220] Key type .llcrypt unregistered [ 616.040759] Key type ._llcrypt unregistered [ 646.140137] Key type ._llcrypt registered [ 646.141765] Key type .llcrypt registered [ 646.227517] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 647.227247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 647.447697] alg: No test for adler32 (adler32-zlib) [ 648.684214] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 649.094648] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 650.903177] Key type lgssc registered [ 652.313764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 660.677094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 660.693649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.175353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.677384] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 672.758848] LustreError: 24260:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.773734] LustreError: 24260:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 676.015669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.343562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 680.657826] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 688.431878] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 694.726264] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 702.852927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 703.437855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.523239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 711.993478] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 721.221498] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 741.462287] Lustre: server umount lustre-OST0000 complete [ 747.491653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 747.500228] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 747.506117] Lustre: Skipped 1 previous similar message [ 747.518182] Lustre: Skipped 1 previous similar message [ 752.617401] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 752.632291] Lustre: Skipped 1 previous similar message [ 756.005239] Lustre: server umount lustre-MDT0000 complete [ 757.747807] LustreError: 24990:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 758.755755] Lustre: 23574:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784938606/real 1784938606] req@0000000024841551 x1871643686237952/t0(0) o13->lustre-OST0000-osc-MDT0001@192.168.201.109@tcp:7/4 lens 224/368 e 0 to 1 dl 1784938622 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 758.824862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 758.879549] LustreError: 25502:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 758.908615] LustreError: 25502:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 760.959279] LustreError: 24240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938624 with bad export cookie 2886483706507932652 [ 760.963305] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 760.970228] LustreError: 24240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 761.382813] Lustre: server umount lustre-MDT0001 complete [ 772.725799] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 775.544668] Key type lgssc unregistered [ 775.833508] LNet: 28239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.843368] LNetError: 28239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.870754] LNet: Removed LNI 192.168.201.109@tcp [ 776.810713] Key type .llcrypt unregistered [ 776.812308] Key type ._llcrypt unregistered [ 804.000950] Key type ._llcrypt registered [ 804.002984] Key type .llcrypt registered [ 804.130586] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 806.171316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 806.231713] alg: No test for adler32 (adler32-zlib) [ 807.229189] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 807.476287] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 809.145660] Key type lgssc registered [ 810.427717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 819.048137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 819.076291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 820.566591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 825.840832] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 830.962090] LustreError: 29655:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.997136] LustreError: 29655:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 836.071457] LustreError: 29656:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 836.594756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.220933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 843.836785] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 853.078624] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 860.147414] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 877.512237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 877.958240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 886.081162] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 890.276268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 894.863349] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 905.718847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 905.740217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 905.747379] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 905.771474] Lustre: Skipped 1 previous similar message [ 909.866868] Lustre: server umount lustre-OST0000 complete [ 915.942828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 915.946839] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 915.953994] Lustre: Skipped 2 previous similar messages [ 921.072409] LustreError: 30574:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 921.107585] LustreError: 30574:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 921.335810] Lustre: server umount lustre-MDT0000 complete [ 925.154633] LustreError: 29650:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 925.199108] LustreError: 29650:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 925.934610] LustreError: 32495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938789 with bad export cookie 15097845374308477636 [ 925.940375] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 925.940741] LustreError: 32495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 926.286944] Lustre: server umount lustre-MDT0001 complete [ 939.099500] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 941.724248] Key type lgssc unregistered [ 942.071609] LNet: 33538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.080161] LNetError: 33538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.114356] LNet: Removed LNI 192.168.201.109@tcp [ 942.991380] Key type .llcrypt unregistered [ 942.994633] Key type ._llcrypt unregistered [ 970.558462] Key type ._llcrypt registered [ 970.560637] Key type .llcrypt registered [ 970.648285] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 972.109080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 972.132184] alg: No test for adler32 (adler32-zlib) [ 973.280615] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 973.545409] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 975.239781] Key type lgssc registered [ 976.584816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 985.592843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 985.611152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.113321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.861635] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 997.352095] LustreError: 34952:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.379435] LustreError: 34952:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 1000.979459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.373345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1005.705738] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1014.528391] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1021.477514] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1028.304190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1028.757914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1032.924395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1036.855271] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1046.735800] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1057.765541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1057.775831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 1057.798132] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 1062.762413] Lustre: server umount lustre-OST0000 complete [ 1073.122115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 1073.143113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1073.179765] Lustre: Skipped 1 previous similar message [ 1073.198964] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 1073.201988] Lustre: Skipped 1 previous similar message [ 1074.599996] Lustre: server umount lustre-MDT0000 complete [ 1077.736210] LustreError: 34951:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.751719] LustreError: 34951:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 1079.644983] LustreError: 34933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938943 with bad export cookie 14847934227151415038 [ 1079.651080] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1079.656985] LustreError: 34933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1080.015802] Lustre: server umount lustre-MDT0001 complete [ 1094.085487] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1097.216820] Key type lgssc unregistered [ 1097.629592] LNet: 38486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.637908] LNetError: 38486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.655198] LNet: Removed LNI 192.168.201.109@tcp [ 1098.677992] Key type .llcrypt unregistered [ 1098.680975] Key type ._llcrypt unregistered [ 1126.757020] Key type ._llcrypt registered [ 1126.758971] Key type .llcrypt registered [ 1126.946377] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1128.246815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1128.257847] alg: No test for adler32 (adler32-zlib) [ 1129.547025] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1129.794968] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1131.512116] Key type lgssc registered [ 1133.183897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1142.354551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1142.381376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.865219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.405389] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1153.006253] LustreError: 39900:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1153.023382] LustreError: 39900:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 1158.118739] LustreError: 39901:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.725791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.436965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1167.202623] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1176.280642] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1183.448243] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1192.154232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1192.480256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1201.542531] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1211.163631] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1213.966388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1227.234684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 1227.242552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1227.256375] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 1227.531837] Lustre: server umount lustre-OST0000 complete [ 1245.665232] Lustre: 39213:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939093/real 1784939093] req@000000009f26b0d5 x1871644189556736/t0(0) o13->lustre-OST0000-osc-MDT0000@192.168.201.109@tcp:7/4 lens 224/368 e 0 to 1 dl 1784939109 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1245.668686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1245.702038] Lustre: 39213:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1250.797897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1250.815884] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 1250.818989] Lustre: Skipped 1 previous similar message [ 1250.837656] Lustre: Skipped 1 previous similar message [ 1255.912390] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 1255.934664] Lustre: Skipped 1 previous similar message [ 1256.923994] Lustre: server umount lustre-MDT0000 complete [ 1261.034680] LustreError: 39901:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.068518] LustreError: 39901:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 1261.216416] LustreError: 39881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939125 with bad export cookie 8180733466169948232 [ 1261.224543] LustreError: 39881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1261.227300] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1267.711991] Lustre: server umount lustre-MDT0001 complete [ 1281.638281] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1285.262498] Key type lgssc unregistered [ 1285.690705] LNet: 43632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.698087] LNetError: 43632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.718339] LNet: Removed LNI 192.168.201.109@tcp [ 1286.768658] Key type .llcrypt unregistered [ 1286.772223] Key type ._llcrypt unregistered [ 1315.169803] Key type ._llcrypt registered [ 1315.171784] Key type .llcrypt registered [ 1315.312957] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1316.645493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1316.783639] alg: No test for adler32 (adler32-zlib) [ 1317.890263] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1318.233222] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1319.975188] Key type lgssc registered [ 1321.329957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1331.211676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1331.245179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.803126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.272752] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1342.953244] LustreError: 45046:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.988537] LustreError: 45046:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 1348.070177] LustreError: 45047:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.115162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.353295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1352.318129] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1361.393101] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1369.926762] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1377.618778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1378.087512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1384.941059] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1392.492786] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1402.611189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1410.037340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 1410.050698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1410.067719] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 1413.102856] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 1413.122923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1413.146517] Lustre: Skipped 1 previous similar message [ 1418.223575] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 1418.232822] Lustre: Skipped 5 previous similar messages [ 1423.327174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1423.376887] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 1423.386993] Lustre: Skipped 3 previous similar messages [ 1423.765783] Lustre: server umount lustre-MDT0000 complete [ 1423.961094] LustreError: 45041:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1428.449550] LustreError: 45042:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1428.998836] LustreError: 45026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939293 with bad export cookie 5655610679163847812 [ 1428.999365] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1429.004471] LustreError: 45026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1429.120217] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 1433.569077] LustreError: 47537:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.571659] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1433.597883] LustreError: 47537:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 1438.689942] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.109@tcp (stopping) [ 1438.702287] Lustre: Skipped 2 previous similar messages [ 1443.295873] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1443.520780] Lustre: server umount lustre-MDT0001 complete [ 1458.763945] Lustre: server umount lustre-OST0000 complete [ 1479.659961] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1483.298181] Key type lgssc unregistered [ 1483.701607] LNet: 48931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1483.721814] LNetError: 48931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.756387] LNet: Removed LNI 192.168.201.109@tcp [ 1484.681272] Key type .llcrypt unregistered [ 1484.684050] Key type ._llcrypt unregistered [ 1511.218274] Key type ._llcrypt registered [ 1511.220457] Key type .llcrypt registered [ 1511.388503] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1512.362914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1512.406743] alg: No test for adler32 (adler32-zlib) [ 1513.671380] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1513.943874] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1515.607780] Key type lgssc registered [ 1517.053688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1524.477705] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1524.502039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1540.063404] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1545.185193] LustreError: 50317:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 1571.881635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.044334] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1585.143477] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1751.238144] Lustre: server umount lustre-OST0000 complete [ 1767.854765] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1770.746891] Key type lgssc unregistered [ 1771.060400] LNet: 51991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.070569] LNetError: 51991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1772.133799] LNet: Removed LNI 192.168.201.109@tcp [ 1773.184325] Key type .llcrypt unregistered [ 1773.187829] Key type ._llcrypt unregistered [ 1795.157987] Key type ._llcrypt registered [ 1795.164260] Key type .llcrypt registered [ 1795.298987] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1796.748220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1796.786820] alg: No test for adler32 (adler32-zlib) [ 1797.871791] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1798.209226] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1799.975452] Key type lgssc registered [ 1801.372529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1809.106548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1809.124385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1810.644367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.562345] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1820.130359] LustreError: 53385:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1820.163404] LustreError: 53385:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 1824.907324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1825.251449] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.109@tcp (not set up) [ 1825.461279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1830.766780] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1839.338467] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1846.139257] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1854.222275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1854.779182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.852602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1862.395233] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1870.727706] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1876.464584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1876.466123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 1876.489107] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 1876.501854] Lustre: Skipped 1 previous similar message [ 1879.758343] Lustre: server umount lustre-OST0000 complete [ 1886.698701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 1886.701472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1886.705681] Lustre: Skipped 1 previous similar message [ 1886.708925] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 1889.839732] Lustre: server umount lustre-MDT0000 complete [ 1891.818069] LustreError: 53386:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1891.849279] LustreError: 53386:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 1895.126087] LustreError: 53365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939759 with bad export cookie 7531846911027384775 [ 1895.150215] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1895.155304] LustreError: 53365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1895.869189] Lustre: server umount lustre-MDT0001 complete [ 1911.362880] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1915.231460] Key type lgssc unregistered [ 1915.592394] LNet: 56922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.603599] LNetError: 56922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.626675] LNet: Removed LNI 192.168.201.109@tcp [ 1916.796353] Key type .llcrypt unregistered [ 1916.798766] Key type ._llcrypt unregistered [ 1940.409624] Key type ._llcrypt registered [ 1940.411229] Key type .llcrypt registered [ 1940.525486] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1941.581880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1941.686195] alg: No test for adler32 (adler32-zlib) [ 1942.848458] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1943.046112] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1944.703182] Key type lgssc registered [ 1945.912741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1954.240769] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1954.263333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1969.887428] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1975.007354] LustreError: 58312:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 2001.631220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.920960] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2016.777957] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2024.398905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.064223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.165391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2030.063326] LustreError: 58336:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.089395] LustreError: 58336:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 2031.096558] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2035.193519] LustreError: 58337:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.202082] LustreError: 58336:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.220336] LustreError: 58336:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2041.282750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.317404] LustreError: 59349:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.346855] LustreError: 59349:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2041.747609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2046.101654] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2054.568635] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2061.333378] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2065.910837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 2065.924314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2065.943952] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2067.427767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 2067.439778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2067.455955] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2067.478877] Lustre: Skipped 1 previous similar message [ 2069.665662] Lustre: server umount lustre-OST0000 complete [ 2093.034625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 2093.047889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2093.073141] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2097.120591] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2097.122792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2098.448583] Lustre: server umount lustre-MDT0000 complete [ 2102.256682] LustreError: 60077:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2102.268751] LustreError: 60077:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2102.448849] LustreError: 58318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939966 with bad export cookie 7385060028499194311 [ 2102.457109] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2102.461100] LustreError: 58318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2108.869795] Lustre: server umount lustre-MDT0001 complete [ 2121.533981] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2124.872305] Key type lgssc unregistered [ 2125.161081] LNet: 62056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.171624] LNetError: 62056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2126.267126] LNet: Removed LNI 192.168.201.109@tcp [ 2127.255871] Key type .llcrypt unregistered [ 2127.257872] Key type ._llcrypt unregistered [ 2151.139457] Key type ._llcrypt registered [ 2151.142413] Key type .llcrypt registered [ 2151.253549] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2152.259243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2152.379578] alg: No test for adler32 (adler32-zlib) [ 2153.401846] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2153.576080] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2155.255328] Key type lgssc registered [ 2156.225626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2164.107475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2164.120177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2165.683430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.347949] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2176.033519] LustreError: 63470:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2176.040065] LustreError: 63470:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2180.989446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.089398] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.109@tcp (not set up) [ 2181.378846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2187.328091] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2199.617531] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2206.640931] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2215.282961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2215.610184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.506311] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2232.506394] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2236.170556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2251.748774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2251.754405] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2251.776120] Lustre: Skipped 1 previous similar message [ 2256.784550] Lustre: server umount lustre-OST0000 complete [ 2261.992127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2262.008670] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2262.014508] Lustre: Skipped 1 previous similar message [ 2262.025927] Lustre: Skipped 2 previous similar messages [ 2267.107722] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2267.115735] Lustre: Skipped 1 previous similar message [ 2271.220309] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2271.227529] Lustre: Skipped 1 previous similar message [ 2275.295184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2275.500985] Lustre: server umount lustre-MDT0000 complete [ 2276.324694] LustreError: 63470:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.332758] LustreError: 63470:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2279.341035] LustreError: 63450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940143 with bad export cookie 11390098872046771171 [ 2279.347990] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.363499] LustreError: 63450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2279.698357] Lustre: server umount lustre-MDT0001 complete [ 2291.338825] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2294.528616] Key type lgssc unregistered [ 2294.766794] LNet: 67029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.771829] LNetError: 67029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.788754] LNet: Removed LNI 192.168.201.109@tcp [ 2295.806513] Key type .llcrypt unregistered [ 2295.808991] Key type ._llcrypt unregistered [ 2323.425360] Key type ._llcrypt registered [ 2323.430823] Key type .llcrypt registered [ 2323.547561] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2325.387515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2325.424073] alg: No test for adler32 (adler32-zlib) [ 2326.617292] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2326.859950] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2328.527223] Key type lgssc registered [ 2330.117130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2339.687562] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 20:43:22 (1784940202) [ 2347.648582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2347.682820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.325845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2353.658847] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2359.785816] LustreError: 68733:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.810772] LustreError: 68733:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2362.847489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.506321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2369.550527] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2380.591464] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2388.219286] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2395.447900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2395.819873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2401.831079] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2409.598629] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2419.401179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2425.300338] Lustre: Failing over lustre-MDT0000 [ 2425.860314] Lustre: server umount lustre-MDT0000 complete [ 2425.987715] LustreError: 68729:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.928077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 2429.929933] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2429.944911] LustreError: 69662:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.009801] Lustre: Skipped 2 previous similar messages [ 2435.048076] LustreError: 68729:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.068154] LustreError: 68729:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 5 previous similar messages [ 2436.418161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.581716] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2436.745309] Lustre: *** cfs_fail_loc=135, val=0*** [ 2436.863582] LustreError: 71844:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2436.877064] LustreError: 71844:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.109@tcp: cfg command failed: rc = -2 [ 2436.891591] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2436.891591] [ 2436.909349] LustreError: MGC192.168.201.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2436.932292] LustreError: 71832:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2436.950688] LustreError: 71832:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2436.979407] LustreError: 71832:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2436.986989] LustreError: 71832:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2437.054247] Lustre: server umount lustre-MDT0000 complete [ 2437.062458] LustreError: 71832:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2440.165503] LustreError: 69661:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.182318] LustreError: 69661:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 2445.507827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2445.728373] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2445.739930] LustreError: 68725:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2446.111727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2446.173810] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2446.423325] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2450.680543] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2451.439770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.201.109@tcp (at 192.168.201.109@tcp) [ 2451.500762] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2451.566168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2459.619349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 2459.624989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2459.636612] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2461.673510] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2461.684138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2465.659148] Lustre: server umount lustre-OST0000 complete [ 2471.908911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 2471.911349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2471.922040] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2471.922051] Lustre: Skipped 1 previous similar message [ 2471.949585] Lustre: Skipped 1 previous similar message [ 2475.885916] Lustre: server umount lustre-MDT0000 complete [ 2477.048932] LustreError: 68733:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.069754] LustreError: 68733:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 6 previous similar messages [ 2480.189684] LustreError: 68714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940344 with bad export cookie 8527608287765146776 [ 2480.198276] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2480.208751] LustreError: 68714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2480.705679] Lustre: server umount lustre-MDT0001 complete [ 2493.347508] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2496.248759] Key type lgssc unregistered [ 2496.756757] LNet: 73625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2496.769069] LNetError: 73625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2496.790137] LNet: Removed LNI 192.168.201.109@tcp [ 2497.816417] Key type .llcrypt unregistered [ 2497.818780] Key type ._llcrypt unregistered [ 2525.857430] Key type ._llcrypt registered [ 2525.861850] Key type .llcrypt registered [ 2526.041943] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2527.354884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2527.464038] alg: No test for adler32 (adler32-zlib) [ 2528.726175] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2529.022580] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2530.705807] Key type lgssc registered [ 2531.939465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2540.009991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2540.035472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.450199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.735626] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2551.783851] LustreError: 75042:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.799655] LustreError: 75042:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2556.081621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.551072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2562.031656] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2570.878518] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2577.493451] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2585.039106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2585.473181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2586.503928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2592.527528] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2600.883447] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2609.020396] Lustre: Failing over lustre-MDT0000 [ 2609.262432] Lustre: server umount lustre-MDT0000 complete [ 2612.204632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2612.228583] Lustre: Skipped 1 previous similar message [ 2612.241617] LustreError: 75037:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.955836] LustreError: 75037:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.015147] LustreError: 75037:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2617.318561] LustreError: 75042:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.379939] LustreError: 75042:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2622.469115] LustreError: 75038:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.496807] LustreError: 75038:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2625.156096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.552986] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2625.777607] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2626.007399] LustreError: 78132:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2626.018819] LustreError: 78132:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.109@tcp: cfg command failed: rc = -9 [ 2626.043040] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2626.043040] [ 2626.070661] LustreError: MGC192.168.201.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2626.087828] LustreError: 78119:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2626.105612] LustreError: 78119:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2626.114555] LustreError: 78119:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2626.120677] LustreError: 78119:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2626.141901] Lustre: server umount lustre-MDT0000 complete [ 2626.148883] LustreError: 78119:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2632.683052] LustreError: 75042:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.723950] LustreError: 75042:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 8 previous similar messages [ 2635.227208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.314032] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2635.325972] LustreError: 75034:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2635.618088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.698383] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2640.342900] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2640.913347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.201.109@tcp (at 192.168.201.109@tcp) [ 2640.993917] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2641.095028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2641.993511] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2651.621709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 2651.634139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2651.662840] Lustre: Skipped 1 previous similar message [ 2651.680971] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2653.163473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 2653.171168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2653.184045] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2653.601840] Lustre: server umount lustre-OST0000 complete [ 2661.358305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2661.359452] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2661.370830] Lustre: Skipped 1 previous similar message [ 2661.386267] Lustre: Skipped 1 previous similar message [ 2664.395340] Lustre: server umount lustre-MDT0000 complete [ 2666.466736] LustreError: 75042:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.489617] LustreError: 75042:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 2668.759150] LustreError: 76887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940532 with bad export cookie 6211908065532494320 [ 2668.760167] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2668.777795] LustreError: 76887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2669.343364] Lustre: server umount lustre-MDT0001 complete [ 2681.374652] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2684.127544] Key type lgssc unregistered [ 2684.494417] LNet: 79914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2684.508636] LNetError: 79914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2685.541675] LNet: Removed LNI 192.168.201.109@tcp [ 2686.595391] Key type .llcrypt unregistered [ 2686.598602] Key type ._llcrypt unregistered [ 2702.037547] Key type ._llcrypt registered [ 2702.043131] Key type .llcrypt registered [ 2702.198604] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2702.648732] Key type .llcrypt unregistered [ 2702.650591] Key type ._llcrypt unregistered [ 2729.513416] Key type ._llcrypt registered [ 2729.518991] Key type .llcrypt registered [ 2729.739306] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2731.778476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.810316] alg: No test for adler32 (adler32-zlib) [ 2732.960859] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2733.323734] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2735.063518] Key type lgssc registered [ 2736.701387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2745.492101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2745.513574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.873265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2752.805878] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2756.085223] LustreError: 81816:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.104913] LustreError: 81816:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2761.194348] LustreError: 81817:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.093203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.504312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2769.432913] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2778.548116] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2785.047432] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2793.775531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2794.411127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.146995] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2810.589138] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2816.975314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2830.313740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 2830.326058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2830.341419] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2832.364978] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2832.373083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2832.394076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 2836.759706] Lustre: server umount lustre-OST0000 complete [ 2842.596620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2842.606125] Lustre: Skipped 1 previous similar message [ 2842.607377] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2842.619738] Lustre: Skipped 2 previous similar messages [ 2847.717616] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2847.731368] Lustre: Skipped 1 previous similar message [ 2848.461405] Lustre: server umount lustre-MDT0000 complete [ 2852.235387] LustreError: 81796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940716 with bad export cookie 2310306205215961951 [ 2852.250015] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2852.258491] LustreError: 81796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2852.762497] Lustre: server umount lustre-MDT0001 complete [ 2865.275821] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2867.226265] Key type lgssc unregistered [ 2867.523525] LNet: 85376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2867.528828] LNetError: 85376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2868.581808] LNet: Removed LNI 192.168.201.109@tcp [ 2869.769196] Key type .llcrypt unregistered [ 2869.773194] Key type ._llcrypt unregistered [ 2899.527243] Key type ._llcrypt registered [ 2899.529408] Key type .llcrypt registered [ 2899.671913] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2901.333292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2901.378027] alg: No test for adler32 (adler32-zlib) [ 2902.460508] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2902.829770] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2904.559246] Key type lgssc registered [ 2905.990483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2914.875988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2914.899575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2916.399059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.356908] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2926.565137] LustreError: 86795:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.591022] LustreError: 86795:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2930.838331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.283623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2936.840483] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2945.437436] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2952.089234] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2960.097794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2960.449153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.479282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2967.285852] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2976.451447] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2986.984831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 2986.994267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2986.997172] LustreError: Skipped 1 previous similar message [ 2986.999687] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2987.016407] Lustre: Skipped 1 previous similar message [ 2989.589353] Lustre: server umount lustre-OST0000 complete [ 2997.225241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2997.232431] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 2997.251220] Lustre: Skipped 1 previous similar message [ 2997.272170] Lustre: Skipped 2 previous similar messages [ 2999.782229] Lustre: server umount lustre-MDT0000 complete [ 3002.339130] LustreError: 86796:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.367648] LustreError: 86796:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 3004.183918] LustreError: 86778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940868 with bad export cookie 18235008582945135470 [ 3004.184455] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3004.203638] LustreError: 86778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3004.792494] Lustre: server umount lustre-MDT0001 complete [ 3018.582550] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3021.801712] Key type lgssc unregistered [ 3022.071337] LNet: 90330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3022.078033] LNetError: 90330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3022.101818] LNet: Removed LNI 192.168.201.109@tcp [ 3023.163962] Key type .llcrypt unregistered [ 3023.166394] Key type ._llcrypt unregistered [ 3049.227685] Key type ._llcrypt registered [ 3049.230775] Key type .llcrypt registered [ 3049.379424] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3050.763441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3050.845317] alg: No test for adler32 (adler32-zlib) [ 3051.904312] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 3052.184437] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3053.895456] Key type lgssc registered [ 3055.258841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3064.085267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3064.121732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.727285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.137583] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3075.048611] LustreError: 91742:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.063834] LustreError: 91742:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 3080.184240] LustreError: 91743:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.368901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.898838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3088.638284] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3097.153263] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3102.992522] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3110.062320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.489277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.577316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3116.356995] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3125.473898] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3140.070872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 192.168.201.109@tcp failed: rc = -107 [ 3140.079808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3140.095629] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3143.666072] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3143.676036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 3143.676199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3148.775238] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3148.783501] Lustre: Skipped 2 previous similar messages [ 3153.375344] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3153.508222] Lustre: server umount lustre-OST0000 complete [ 3159.029394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3159.039382] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3159.046685] Lustre: Skipped 1 previous similar message [ 3163.541978] Lustre: server umount lustre-MDT0000 complete [ 3164.134701] LustreError: 91743:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.156358] LustreError: 91743:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 3166.966855] LustreError: 91724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941030 with bad export cookie 4992127365350858414 [ 3166.985964] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3167.309497] Lustre: server umount lustre-MDT0001 complete [ 3178.662295] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3181.360700] Key type lgssc unregistered [ 3181.654269] LNet: 95303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3181.671418] LNetError: 95303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3181.683683] LNet: Removed LNI 192.168.201.109@tcp [ 3182.399915] Key type .llcrypt unregistered [ 3182.403029] Key type ._llcrypt unregistered [ 3207.665872] Key type ._llcrypt registered [ 3207.670356] Key type .llcrypt registered [ 3207.824860] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3209.500691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3209.567478] alg: No test for adler32 (adler32-zlib) [ 3210.780970] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 3211.120575] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3212.887168] Key type lgssc registered [ 3214.343298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3223.000756] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3223.018771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3238.560265] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3243.679371] LustreError: 96690:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 3270.367276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.322253] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3286.686952] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3293.579852] Lustre: server umount lustre-OST0000 complete [ 3304.270224] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 20:59:26 (1784941166) [ 3312.893930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3313.442912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3318.666531] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3318.762945] LustreError: 98328:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3318.797234] LustreError: 98328:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 3323.883466] LustreError: 98329:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3328.247399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.729778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3334.066531] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3343.813628] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3350.541601] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3358.595862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3358.709791] LustreError: 100150:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3358.970178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.364590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3366.595751] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3375.702921] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3388.905075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 3388.910675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3388.941186] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3388.947521] Lustre: Skipped 1 previous similar message [ 3393.846767] Lustre: server umount lustre-OST0000 complete [ 3399.156371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3399.159358] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3399.172043] Lustre: Skipped 2 previous similar messages [ 3399.189719] Lustre: Skipped 1 previous similar message [ 3404.263421] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3404.278435] Lustre: Skipped 1 previous similar message [ 3404.587399] Lustre: server umount lustre-MDT0000 complete [ 3408.945388] LustreError: 98309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941272 with bad export cookie 4267394987006040530 [ 3408.962019] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3409.397357] Lustre: server umount lustre-MDT0001 complete [ 3423.259930] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3426.034742] Key type lgssc unregistered [ 3426.433296] LNet: 102034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3426.439629] LNetError: 102034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3426.461130] LNet: Removed LNI 192.168.201.109@tcp [ 3428.025575] Key type .llcrypt unregistered [ 3428.028679] Key type ._llcrypt unregistered [ 3478.929981] Key type ._llcrypt registered [ 3478.935375] Key type .llcrypt registered [ 3479.072144] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3480.906299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3480.940323] alg: No test for adler32 (adler32-zlib) [ 3482.236567] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 3482.615145] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3484.375395] Key type lgssc registered [ 3486.041495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3495.774849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3495.792453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.383901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3502.569652] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3507.683691] LustreError: 104228:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3507.699714] LustreError: 104228:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 3512.432151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3512.869671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3518.409100] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3527.210254] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3533.495911] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3540.947349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3541.392498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3542.480456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3548.978908] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3557.470404] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3568.099293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 3568.114607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3568.124960] LustreError: Skipped 1 previous similar message [ 3568.127480] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3568.145141] Lustre: Skipped 1 previous similar message [ 3568.167863] Lustre: Skipped 1 previous similar message [ 3572.182216] Lustre: server umount lustre-OST0000 complete [ 3578.356278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3578.362601] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3578.372139] Lustre: Skipped 1 previous similar message [ 3582.915299] Lustre: server umount lustre-MDT0000 complete [ 3583.458615] LustreError: 104224:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3583.476241] LustreError: 104224:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 3586.821774] LustreError: 104208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941450 with bad export cookie 6124215172631662244 [ 3586.827877] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3586.833636] LustreError: 104208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3587.178356] Lustre: server umount lustre-MDT0001 complete [ 3599.962678] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3603.491949] Key type lgssc unregistered [ 3603.831773] LNet: 107763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3603.840305] LNetError: 107763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3604.901954] LNet: Removed LNI 192.168.201.109@tcp [ 3605.655570] Key type .llcrypt unregistered [ 3605.658985] Key type ._llcrypt unregistered [ 3624.560821] Key type ._llcrypt registered [ 3624.562637] Key type .llcrypt registered [ 3624.652293] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3625.396389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3625.520599] alg: No test for adler32 (adler32-zlib) [ 3626.605806] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 3626.855099] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3628.511218] Key type lgssc registered [ 3629.797373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3637.281944] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3637.293133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3652.831488] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3657.951316] LustreError: 109006:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 3684.575363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.426843] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3698.774507] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3705.721557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3705.974323] LustreError: MGC192.168.201.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3705.987930] LustreError: 110028:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3705.997801] LustreError: 110028:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3706.000297] LustreError: 110028:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3706.112291] Lustre: server umount lustre-MDT0000 complete [ 3706.113880] LustreError: 110028:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3723.406506] Lustre: server umount lustre-OST0000 complete [ 3734.633756] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3745.355479] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3784.110566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.422838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3784.485667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3784.610566] Lustre: lustre-MDT0000: new disk, initializing [ 3784.734286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3784.774536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3789.869099] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3800.325578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3800.482299] Lustre: 112820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3800.542212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3800.548934] Lustre: Skipped 1 previous similar message [ 3800.669403] Lustre: lustre-MDT0001: new disk, initializing [ 3800.753743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3800.781468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3800.793676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3805.309217] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3814.086182] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3819.421779] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3825.229077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3825.426091] Lustre: lustre-OST0000: new disk, initializing [ 3825.428973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3825.437313] Lustre: 114786:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3825.509142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3830.890303] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3832.424365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3832.441172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3832.554042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3839.745948] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3844.363420] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3844.592283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3848.657080] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3848.898107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3853.301586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3853.304362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 3853.307787] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3853.320385] Lustre: Skipped 1 previous similar message [ 3853.333610] LustreError: Skipped 1 previous similar message [ 3856.589794] Lustre: server umount lustre-OST0000 complete [ 3862.497264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 3862.508329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3862.535745] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3862.550713] Lustre: Skipped 1 previous similar message [ 3863.522076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3863.542923] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3863.554477] Lustre: Skipped 1 previous similar message [ 3865.561943] Lustre: server umount lustre-MDT0000 complete [ 3868.406062] LustreError: 112811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941732 with bad export cookie 5369044855639854955 [ 3868.412944] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3868.415542] LustreError: 112811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3868.692875] Lustre: server umount lustre-MDT0001 complete [ 3875.413705] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 21:08:58 (1784941738) [ 3878.378494] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3892.483380] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3899.803911] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3929.338817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3929.537591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3929.557293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3929.601284] Lustre: lustre-MDT0000: new disk, initializing [ 3929.662801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3929.675620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3933.697251] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3942.634629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3942.711523] Lustre: 119683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3942.748249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3942.751691] Lustre: Skipped 1 previous similar message [ 3942.797549] Lustre: lustre-MDT0001: new disk, initializing [ 3942.864991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3942.888660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3942.901956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3946.828734] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3954.530741] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3959.058800] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3964.545183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3964.702144] Lustre: lustre-OST0000: new disk, initializing [ 3964.705993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3964.710059] Lustre: 121649:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3964.757734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.276408] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3972.129749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3972.137438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3972.241470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3976.654295] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3980.594460] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3980.755800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3984.897300] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3985.089177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3987.425863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 3987.434992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3987.443794] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3992.813072] Lustre: server umount lustre-OST0000 complete [ 3998.180552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3998.182857] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 3998.188415] Lustre: Skipped 2 previous similar messages [ 3998.193228] Lustre: Skipped 1 previous similar message [ 4002.583767] Lustre: server umount lustre-MDT0000 complete [ 4005.978704] LustreError: 119674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941869 with bad export cookie 5369044855639855837 [ 4005.981748] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4005.990367] LustreError: 119674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4006.230977] Lustre: server umount lustre-MDT0001 complete [ 4014.051836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4014.522818] LustreError: 123866:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4014.541850] LustreError: 123866:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 4014.606365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4018.811861] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4019.682250] LustreError: 123867:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4024.804573] LustreError: 123866:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4025.214909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4029.740929] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4035.776449] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4040.553479] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4046.197921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4051.619431] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4057.517669] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4067.472995] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4071.394641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 4071.398199] LustreError: Skipped 1 previous similar message [ 4071.403227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4071.413530] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 4075.499791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 4076.906573] Lustre: server umount lustre-OST0000 complete [ 4086.288065] Lustre: server umount lustre-MDT0000 complete [ 4089.258882] LustreError: 126745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941953 with bad export cookie 5369044855639856719 [ 4089.262162] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4089.265238] LustreError: 126745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4089.534979] Lustre: server umount lustre-MDT0001 complete [ 4098.031968] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4100.084530] Key type lgssc unregistered [ 4100.277775] LNet: 127497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4100.282683] LNetError: 127497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4101.354744] LNet: Removed LNI 192.168.201.109@tcp [ 4101.968149] Key type .llcrypt unregistered [ 4101.970064] Key type ._llcrypt unregistered [ 4114.300384] Key type ._llcrypt registered [ 4114.302231] Key type .llcrypt registered [ 4114.389539] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4125.651351] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4126.324093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4126.440062] alg: No test for adler32 (adler32-zlib) [ 4127.416203] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 4127.564276] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4129.207169] Key type lgssc registered [ 4130.026722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4155.362758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4155.390898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4156.581313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4156.621048] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4156.685319] Lustre: lustre-MDT0000: new disk, initializing [ 4156.741212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4156.756056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4160.298228] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4168.232037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4168.324373] Lustre: 130630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4168.361560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4168.365248] Lustre: Skipped 1 previous similar message [ 4168.417111] Lustre: lustre-MDT0001: new disk, initializing [ 4168.479366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4168.521733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4168.531409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4171.425647] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4177.340642] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4181.539914] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4186.573293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4186.819630] Lustre: lustre-OST0000: new disk, initializing [ 4186.824125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4186.829415] Lustre: 132594:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4186.898974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4191.743983] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4193.317445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4193.325864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4193.379301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4199.114844] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4202.202246] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4202.352846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4205.344053] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4205.468626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4208.614980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4208.616357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 4208.619993] Lustre: Skipped 1 previous similar message [ 4208.621631] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 4208.630209] LustreError: Skipped 1 previous similar message [ 4208.636167] Lustre: Skipped 1 previous similar message [ 4213.119881] Lustre: server umount lustre-OST0000 complete [ 4218.851271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4218.859430] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 4218.862501] Lustre: Skipped 1 previous similar message [ 4218.879463] Lustre: Skipped 1 previous similar message [ 4222.485177] Lustre: server umount lustre-MDT0000 complete [ 4225.172368] LustreError: 130622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942089 with bad export cookie 4484035150852412568 [ 4225.176039] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4225.183544] LustreError: 130622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4225.411070] Lustre: server umount lustre-MDT0001 complete [ 4231.901539] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 21:14:55 (1784942095) [ 4236.261151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.489726] LustreError: 135004:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.501906] LustreError: 135004:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 4236.533484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4239.301990] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4241.893220] LustreError: 135005:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.910800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4245.191312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4247.944434] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4253.097199] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4256.616827] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4260.837470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4260.840115] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 4260.843432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 4260.849972] Lustre: Skipped 1 previous similar message [ 4260.860778] Lustre: Skipped 1 previous similar message [ 4264.247457] Lustre: server umount lustre-MDT0000 complete [ 4265.952547] LustreError: 135812:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4265.969798] LustreError: 135812:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 4266.369441] LustreError: 134984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942130 with bad export cookie 4484035150852413457 [ 4266.373861] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4266.377482] LustreError: 134984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4266.540986] Lustre: server umount lustre-MDT0001 complete [ 4271.440763] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 21:15:34 (1784942134) [ 4275.278837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4290.784330] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4295.903510] LustreError: 137416:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 4321.632768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.972923] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4331.510933] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4333.023492] Lustre: server umount lustre-OST0000 complete [ 4338.737910] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 21:16:41 (1784942201) [ 4343.397506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4343.612457] LustreError: 138858:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4343.660626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4346.568431] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4348.896872] LustreError: 138859:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4351.697334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4354.264176] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4358.612860] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4362.057383] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4366.676644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4370.818151] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4375.552197] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4381.162753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4381.165325] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 4381.173068] Lustre: Skipped 2 previous similar messages [ 4381.187758] Lustre: Skipped 2 previous similar messages [ 4386.274988] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 4386.286969] Lustre: Skipped 2 previous similar messages [ 4386.576929] Lustre: server umount lustre-MDT0000 complete [ 4389.821892] LustreError: 138841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942253 with bad export cookie 4484035150852414122 [ 4389.828438] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4389.832611] LustreError: 138841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4389.984114] Lustre: server umount lustre-MDT0001 complete [ 4402.754439] Lustre: server umount lustre-OST0000 complete [ 4408.582290] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 21:17:51 (1784942271) [ 4412.681360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4412.961491] LustreError: 142507:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.012811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4413.016773] Lustre: Skipped 2 previous similar messages [ 4415.881086] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4421.750036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4424.695485] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4429.255340] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4432.721368] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4436.915497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4438.190919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4440.997387] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4445.439766] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4447.823348] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4447.950926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4450.399426] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4450.558541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4451.810223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4451.821437] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 4451.823907] Lustre: Skipped 1 previous similar message [ 4451.824403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 4451.833427] Lustre: Skipped 1 previous similar message [ 4457.892582] Lustre: server umount lustre-OST0000 complete [ 4462.049890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4462.057199] Lustre: Skipped 1 previous similar message [ 4466.404952] Lustre: server umount lustre-MDT0000 complete [ 4467.169422] LustreError: 143235:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4467.197053] LustreError: 143235:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 4469.244795] LustreError: 142489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942333 with bad export cookie 4484035150852415172 [ 4469.248370] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4469.255722] LustreError: 142489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4469.523033] Lustre: server umount lustre-MDT0001 complete [ 4475.295181] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 21:18:58 (1784942338) [ 4480.571562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4496.159842] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4501.279524] LustreError: 146663:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 4527.007318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4527.014144] Lustre: Skipped 2 previous similar messages [ 4531.573359] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4536.565695] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4541.141582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4541.468944] LustreError: 146688:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.584393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4544.736029] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4549.960673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4552.473979] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4557.423937] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4561.341960] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4564.401904] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4564.594519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4567.257569] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4567.400271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4570.601459] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4570.605170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 4570.606150] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 4570.606159] Lustre: Skipped 4 previous similar messages [ 4570.607372] Lustre: Skipped 1 previous similar message [ 4574.646716] Lustre: server umount lustre-MDT0000 complete [ 4576.654746] LustreError: 146671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942440 with bad export cookie 4484035150852416187 [ 4576.659428] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4576.660108] LustreError: 146671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4576.808791] Lustre: server umount lustre-MDT0001 complete [ 4588.091362] Lustre: server umount lustre-OST0000 complete [ 4593.196965] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 21:20:56 (1784942456) [ 4598.302996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4613.856821] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4618.975636] LustreError: 150853:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 4647.759572] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4652.501942] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4656.740606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.013309] LustreError: 150878:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.024260] LustreError: 150878:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 7 previous similar messages [ 4657.073756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4657.079739] Lustre: Skipped 3 previous similar messages [ 4657.118119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4659.838503] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4665.393062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.108752] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4672.820315] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4676.620080] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4681.113199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4681.218809] Lustre: lustre-OST0001: new disk, initializing [ 4681.220496] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4681.224269] Lustre: 153718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4683.227785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4683.237335] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4683.310312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4684.965071] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4690.954987] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4693.317672] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4693.415317] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4696.259932] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4696.411555] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4698.081667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 4698.094586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4698.104917] Lustre: Skipped 1 previous similar message [ 4698.108444] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 4698.112921] Lustre: Skipped 2 previous similar messages [ 4703.959966] Lustre: server umount lustre-OST0000 complete [ 4708.844364] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 4708.849673] LustreError: Skipped 1 previous similar message [ 4712.176552] Lustre: server umount lustre-OST0001 complete [ 4720.614771] Lustre: server umount lustre-MDT0000 complete [ 4723.148554] LustreError: 150860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942587 with bad export cookie 4484035150852417139 [ 4723.152225] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4723.157898] LustreError: 150860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4723.345460] Lustre: server umount lustre-MDT0001 complete [ 4737.204269] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 21:23:20 (1784942600) [ 4738.048819] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4738.991640] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4739.963855] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 21:23:23 (1784942603) [ 4743.422084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4743.604445] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4743.623422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4745.902685] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4750.400799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.467540] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4750.490654] Lustre: 157275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4752.539632] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4756.590241] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4759.390782] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4762.544244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4762.656079] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4765.205491] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4768.611449] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4770.969963] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4772.936141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4773.150765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4775.197364] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4775.325687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4777.957119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 4777.958657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4777.960294] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 4777.960300] Lustre: Skipped 8 previous similar messages [ 4777.966800] LustreError: Skipped 1 previous similar message [ 4777.975379] Lustre: Skipped 6 previous similar messages [ 4782.346502] Lustre: server umount lustre-OST0000 complete [ 4798.544865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.640133] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4805.548989] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4808.314723] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4818.410489] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.426444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.428854] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.109@tcp (at 192.168.201.109@tcp) [ 4818.446647] Lustre: Skipped 1 previous similar message [ 4818.855647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 4821.073866] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4821.167950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4824.114148] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff91d4438aa000.ost_server_uuid 50 [ 4825.022921] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91d4438aa000.ost_server_uuid in FULL state after 0 sec [ 4825.152495] LustreError: 158025:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.9@tcp) returned error from blocking AST (req@00000000cd1307f3 x1871647333375360 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000087b5c304/0x3e3a7e3656ec6fbf lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.9@tcp remote: 0xba6681f8dd63276f expref: 5 pid: 158025 timeout: 4925 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4825.169822] LustreError: lustre-MDT0000: A client on nid 192.168.201.9@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4825.174035] LustreError: 157272:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.9@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000087b5c304/0x3e3a7e3656ec6fbf lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.9@tcp remote: 0xba6681f8dd63276f expref: 6 pid: 158025 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4828.648468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 4828.655919] LustreError: Skipped 2 previous similar messages [ 4833.322658] Lustre: server umount lustre-OST0000 complete [ 4841.532453] Lustre: server umount lustre-MDT0000 complete [ 4843.480167] LustreError: 157269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942707 with bad export cookie 4484035150852418322 [ 4843.483421] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4843.485093] LustreError: 157269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4849.664187] Lustre: server umount lustre-MDT0001 complete [ 4855.679648] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4857.254765] Key type lgssc unregistered [ 4857.424716] LNet: 163468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4857.435394] LNetError: 163468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4857.444625] LNet: Removed LNI 192.168.201.109@tcp [ 4857.918377] Key type .llcrypt unregistered [ 4857.919809] Key type ._llcrypt unregistered [ 4872.356961] Key type ._llcrypt registered [ 4872.358550] Key type .llcrypt registered [ 4872.418981] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4873.034633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4873.091602] alg: No test for adler32 (adler32-zlib) [ 4874.023780] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 4874.215350] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4875.855313] Key type lgssc registered [ 4876.537145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4880.829156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4880.845476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4882.110370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4884.719366] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4889.969804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.136366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4892.442159] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4896.770181] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4899.985971] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4904.081118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4904.289953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4907.389231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4907.930383] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4912.206076] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4920.905478] Lustre: Failing over lustre-MDT0000 [ 4921.031674] Lustre: server umount lustre-MDT0000 complete [ 4925.930441] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4925.932420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 4925.932802] LustreError: 166151:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4925.932809] LustreError: 166151:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 4925.945401] Lustre: Skipped 2 previous similar messages [ 4931.041144] LustreError: 164859:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4931.048899] LustreError: 164859:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 4932.379781] LustreError: 164840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.9@tcp arrived at 1784942796 with bad export cookie 7191782455798334335 [ 4932.392037] LustreError: 164840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4935.880645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4935.950419] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4936.070268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.115096] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4938.036386] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4939.003695] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4939.006626] Lustre: lustre-MDT0000: Denying connection for new client e7faee01-c05e-4822-bae8-267ee8c14430 (at 192.168.201.9@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 4941.293078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.201.109@tcp (at 192.168.201.109@tcp) [ 4946.402493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4946.405863] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 4946.408740] Lustre: Skipped 1 previous similar message [ 4946.409733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 4946.411798] Lustre: Skipped 1 previous similar message [ 4951.520693] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 4951.524691] Lustre: Skipped 1 previous similar message [ 4952.481170] Lustre: server umount lustre-OST0000 complete [ 4954.503809] LustreError: 168586:0:(ldlm_lib.c:3026:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4954.508171] Lustre: 167935:0:(ldlm_lib.c:2426:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4954.513722] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4954.525255] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4954.534862] LustreError: 167935:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000025c8751f x1871648116479744/t0(0) o700->lustre-MDT0001-osp-MDT0000@192.168.201.109@tcp:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4954.545315] LustreError: 167935:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4954.550560] LustreError: 167935:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4954.555051] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 4954.757503] Lustre: server umount lustre-MDT0000 complete [ 4956.644063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4956.652264] LustreError: 164854:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.753198] LustreError: 166699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942820 with bad export cookie 7191782455798334440 [ 4956.757265] LustreError: 166699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4956.757816] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4963.001580] Lustre: server umount lustre-MDT0001 complete [ 4967.890709] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4969.250355] Key type lgssc unregistered [ 4969.404777] LNet: 169337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4969.410366] LNetError: 169337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4969.424220] LNet: Removed LNI 192.168.201.109@tcp [ 4969.837087] Key type .llcrypt unregistered [ 4969.839418] Key type ._llcrypt unregistered [ 4982.833894] Key type ._llcrypt registered [ 4982.835096] Key type .llcrypt registered [ 4982.896601] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4983.244611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4983.256953] alg: No test for adler32 (adler32-zlib) [ 4984.146143] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 4984.298365] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4985.895184] Key type lgssc registered [ 4986.460722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4990.658634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4990.676766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4991.890103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4995.294774] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5001.224037] LustreError: 170728:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5001.284839] LustreError: 170728:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 5005.410152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5006.204284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5013.805707] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5030.739088] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5037.795760] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5048.514606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5049.201429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5058.883587] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5061.555909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5066.283953] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5071.848500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5071.856187] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5071.862287] Lustre: Skipped 1 previous similar message [ 5071.872394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 5076.965878] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5076.977418] Lustre: Skipped 2 previous similar messages [ 5077.754752] Lustre: server umount lustre-OST0000 complete [ 5082.093140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5082.102626] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5082.117912] Lustre: Skipped 1 previous similar message [ 5082.135780] Lustre: Skipped 1 previous similar message [ 5086.907394] Lustre: server umount lustre-MDT0000 complete [ 5087.202416] LustreError: 170728:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5087.214567] LustreError: 170728:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 5089.849469] LustreError: 170708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942953 with bad export cookie 13384863990306670442 [ 5089.850086] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5089.863226] LustreError: 170708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5090.175919] Lustre: server umount lustre-MDT0001 complete [ 5098.832989] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5100.946434] Key type lgssc unregistered [ 5101.160512] LNet: 174263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5101.163547] LNetError: 174263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5102.183476] LNet: Removed LNI 192.168.201.109@tcp [ 5102.916204] Key type .llcrypt unregistered [ 5102.918846] Key type ._llcrypt unregistered [ 5131.504900] Key type ._llcrypt registered [ 5131.507736] Key type .llcrypt registered [ 5131.602652] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5132.623599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5132.667365] alg: No test for adler32 (adler32-zlib) [ 5133.677329] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5133.911946] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5135.551197] Key type lgssc registered [ 5136.454314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5143.512402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5143.534912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5144.984175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5149.009849] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5154.284282] LustreError: 175966:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5154.310759] LustreError: 175966:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 5156.743896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.989588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5160.618000] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5167.199771] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5171.801875] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5177.545584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5178.047764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.970615] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5189.397552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5191.355827] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5199.845803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 5199.848101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5199.859699] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5199.868658] Lustre: Skipped 1 previous similar message [ 5203.811903] Lustre: server umount lustre-OST0000 complete [ 5208.035331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 5208.039135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5208.046841] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5208.049793] Lustre: Skipped 1 previous similar message [ 5210.081056] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5210.082340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5212.828979] Lustre: server umount lustre-MDT0000 complete [ 5215.202203] LustreError: 176695:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5215.216273] LustreError: 176695:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 5216.049613] LustreError: 175948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943080 with bad export cookie 5903449546008071119 [ 5216.059213] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5216.068148] LustreError: 175948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5216.321087] Lustre: server umount lustre-MDT0001 complete [ 5226.626654] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5228.917351] Key type lgssc unregistered [ 5229.194749] LNet: 179498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5229.209310] LNetError: 179498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5229.227092] LNet: Removed LNI 192.168.201.109@tcp [ 5230.310993] Key type .llcrypt unregistered [ 5230.313242] Key type ._llcrypt unregistered [ 5248.185622] Key type ._llcrypt registered [ 5248.189291] Key type .llcrypt registered [ 5248.291642] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5248.956815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5248.991365] alg: No test for adler32 (adler32-zlib) [ 5250.050608] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5250.302514] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5251.951167] Key type lgssc registered [ 5253.024386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5262.083727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5262.109332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5263.433148] Lustre: *** cfs_fail_loc=135, val=0*** [ 5263.494400] LustreError: 181009:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5263.499156] LustreError: 181009:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.109@tcp: cfg command failed: rc = -2 [ 5263.506621] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5263.506621] [ 5263.514536] LustreError: MGC192.168.201.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5263.526990] LustreError: 180969:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5263.534850] LustreError: 180969:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5263.544212] LustreError: 180969:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5263.549938] LustreError: 180969:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5263.601341] Lustre: server umount lustre-MDT0000 complete [ 5263.605202] LustreError: 180969:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5269.081800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5269.383130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5273.385465] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5279.722956] LustreError: 181379:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5279.739609] LustreError: 181379:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 5281.088339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5281.439426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5285.443381] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5291.865889] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5297.033527] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5299.169929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5299.180322] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5299.194288] Lustre: Skipped 1 previous similar message [ 5299.208677] Lustre: Skipped 1 previous similar message [ 5304.290177] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5304.298401] Lustre: Skipped 1 previous similar message [ 5305.112472] Lustre: server umount lustre-MDT0000 complete [ 5307.379243] LustreError: 181379:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.394111] LustreError: 181379:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 5308.359897] LustreError: 181361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943172 with bad export cookie 14649548454409095622 [ 5308.362869] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5308.369978] LustreError: 181361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5308.674379] Lustre: server umount lustre-MDT0001 complete [ 5318.545484] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5320.837897] Key type lgssc unregistered [ 5321.134345] LNet: 183702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5321.142261] LNetError: 183702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5321.159361] LNet: Removed LNI 192.168.201.109@tcp [ 5321.823953] Key type .llcrypt unregistered [ 5321.826793] Key type ._llcrypt unregistered [ 5339.075852] Key type ._llcrypt registered [ 5339.077358] Key type .llcrypt registered [ 5339.133963] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5339.310525] Key type .llcrypt unregistered [ 5339.312254] Key type ._llcrypt unregistered [ 5355.750822] Key type ._llcrypt registered [ 5355.752307] Key type .llcrypt registered [ 5355.820523] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5356.841312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5356.910236] alg: No test for adler32 (adler32-zlib) [ 5357.821448] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5357.987355] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5359.615449] Key type lgssc registered [ 5360.556920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5366.750975] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5366.765516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5382.304831] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5387.424768] LustreError: 185832:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 5414.048146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5418.760377] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5424.730962] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5429.457485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5429.955719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5430.006279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5433.021051] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5435.372301] LustreError: 185858:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5435.389956] LustreError: 185858:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 5439.756304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5440.004869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5443.402952] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5449.053839] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5453.617658] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5455.807567] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5465.575739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 5465.581692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5465.599811] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5466.080723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 5466.084397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5468.650582] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5468.662278] Lustre: Skipped 2 previous similar messages [ 5470.558368] Lustre: server umount lustre-OST0000 complete [ 5473.767256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5473.777773] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5473.786101] Lustre: Skipped 1 previous similar message [ 5478.881954] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5478.890649] Lustre: Skipped 2 previous similar messages [ 5479.911806] Lustre: server umount lustre-MDT0000 complete [ 5482.822289] LustreError: 185839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943346 with bad export cookie 16843493893590457314 [ 5482.825085] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5482.830165] LustreError: 185839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5483.095818] Lustre: server umount lustre-MDT0001 complete [ 5491.737205] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5493.621549] Key type lgssc unregistered [ 5493.832917] LNet: 189797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5493.839733] LNetError: 189797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5493.854839] LNet: Removed LNI 192.168.201.109@tcp [ 5494.512585] Key type .llcrypt unregistered [ 5494.515197] Key type ._llcrypt unregistered [ 5514.103217] Key type ._llcrypt registered [ 5514.105308] Key type .llcrypt registered [ 5514.233232] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5514.934589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5515.075506] alg: No test for adler32 (adler32-zlib) [ 5516.111559] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5516.284180] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5517.991344] Key type lgssc registered [ 5518.936130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5525.056906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5525.079589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5526.425921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5530.035279] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5536.743148] LustreError: 191190:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5536.757883] LustreError: 191190:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 5537.087603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.341505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5540.551977] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5545.994858] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5550.477525] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5555.604215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5555.886652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5560.346535] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5565.715806] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5571.269053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5578.923651] Lustre: Failing over lustre-MDT0000 [ 5581.124722] Lustre: server umount lustre-MDT0000 complete [ 5581.416330] LustreError: 191187:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5581.805536] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5581.817804] Lustre: Skipped 1 previous similar message [ 5586.520813] LustreError: 191187:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5586.537084] LustreError: 191187:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 5591.636739] LustreError: 191186:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5591.649256] LustreError: 191186:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 5596.765698] LustreError: 192532:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5596.778919] LustreError: 192532:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 5597.152652] Lustre: 190503:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784943445/real 1784943445] req@00000000fe849389 x1871648789652480/t0(0) o400->MGC192.168.201.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784943461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5597.167880] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5597.176672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.999822] LustreError: 192532:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.017825] LustreError: 192532:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 8 previous similar messages [ 5607.404315] LustreError: 190499:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a7dbc2b3 x1871648789657984/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5607.631091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5607.682387] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5610.010580] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5610.694655] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5613.047260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.201.109@tcp (at 192.168.201.109@tcp) [ 5613.062381] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5613.095170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5613.714514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5618.237938] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5628.392095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5628.398593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 5628.399211] Lustre: Skipped 1 previous similar message [ 5628.402049] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5628.408379] Lustre: Skipped 1 previous similar message [ 5631.898235] Lustre: server umount lustre-OST0000 complete [ 5638.625548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5638.627729] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5638.628825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 5638.632267] Lustre: Skipped 2 previous similar messages [ 5638.645484] Lustre: Skipped 1 previous similar message [ 5640.791209] Lustre: server umount lustre-MDT0000 complete [ 5643.186156] LustreError: 191170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943507 with bad export cookie 9847363831336052862 [ 5643.195853] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5643.421790] Lustre: server umount lustre-MDT0001 complete [ 5651.320329] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5653.232145] Key type lgssc unregistered [ 5653.420763] LNet: 196046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5653.429919] LNetError: 196046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5653.439578] LNet: Removed LNI 192.168.201.109@tcp [ 5653.932882] Key type .llcrypt unregistered [ 5653.934149] Key type ._llcrypt unregistered [ 5670.117351] Key type ._llcrypt registered [ 5670.119607] Key type .llcrypt registered [ 5670.198970] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5671.074577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5671.096093] alg: No test for adler32 (adler32-zlib) [ 5672.136864] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5672.376622] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5674.023577] Key type lgssc registered [ 5675.026850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5680.336721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5680.354259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.668144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5684.811360] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5691.265514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.577834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5694.639168] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5699.958259] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5703.859083] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5708.791976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5709.088645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.815259] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5718.704824] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5727.540146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5731.745229] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5737.946663] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5753.215774] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5762.531049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 5762.534567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5762.541203] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5763.557143] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5763.560567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 5763.575047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5764.917073] Lustre: server umount lustre-OST0000 complete [ 5768.673909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5768.678744] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5768.683205] Lustre: Skipped 1 previous similar message [ 5768.685269] Lustre: Skipped 1 previous similar message [ 5773.794029] LustreError: 198165:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.810256] LustreError: 198165:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 5775.608373] Lustre: server umount lustre-MDT0000 complete [ 5777.958872] LustreError: 197418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943641 with bad export cookie 5929674201917480428 [ 5777.959771] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5777.967305] LustreError: 197418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5778.191681] Lustre: server umount lustre-MDT0001 complete [ 5785.367649] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5787.505460] Key type lgssc unregistered [ 5787.781122] LNet: 201322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5787.784988] LNetError: 201322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5787.794988] LNet: Removed LNI 192.168.201.109@tcp [ 5788.430405] Key type .llcrypt unregistered [ 5788.433538] Key type ._llcrypt unregistered [ 5800.399491] Key type ._llcrypt registered [ 5800.404124] Key type .llcrypt registered [ 5800.498640] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5801.210321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5801.246505] alg: No test for adler32 (adler32-zlib) [ 5802.207681] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5802.395163] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5804.031198] Key type lgssc registered [ 5804.764591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5809.693807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5809.713448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5810.971645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5813.543832] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5818.850517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5819.070136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5821.401076] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5826.237865] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5829.630631] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5833.959549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5834.186938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5835.252692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5837.869656] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5842.595236] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5847.375986] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5855.656686] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5865.898982] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5873.893154] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5881.316097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 5881.323837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5881.338255] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5885.415798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 5885.415861] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5885.417972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5885.429022] Lustre: Skipped 1 previous similar message [ 5886.643501] Lustre: server umount lustre-OST0000 complete [ 5890.531195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5890.531908] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5890.536682] Lustre: Skipped 1 previous similar message [ 5894.818232] Lustre: server umount lustre-MDT0000 complete [ 5895.649529] LustreError: 202565:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5895.670500] LustreError: 202565:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 5896.732434] LustreError: 202546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943760 with bad export cookie 14678342797672547482 [ 5896.738444] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5896.740862] LustreError: 202546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5896.934868] Lustre: server umount lustre-MDT0001 complete [ 5902.514680] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5903.798426] Key type lgssc unregistered [ 5903.976351] LNet: 207776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5903.979235] LNetError: 207776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5903.991498] LNet: Removed LNI 192.168.201.109@tcp [ 5904.380393] Key type .llcrypt unregistered [ 5904.381912] Key type ._llcrypt unregistered [ 5917.569719] Key type ._llcrypt registered [ 5917.571818] Key type .llcrypt registered [ 5917.639636] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5918.290251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5918.310040] alg: No test for adler32 (adler32-zlib) [ 5919.253881] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5919.384150] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5920.983255] Key type lgssc registered [ 5921.599422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5925.708364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5925.718853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.934804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5929.319535] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5933.935504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5934.103425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5936.285638] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5940.262666] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5943.115659] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5946.566701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5946.783497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.069268] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5951.916379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5954.683387] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5964.258562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 5964.258992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5964.268262] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5968.886657] Lustre: server umount lustre-OST0000 complete [ 5974.498953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5974.504653] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5974.508590] Lustre: Skipped 2 previous similar messages [ 5974.521488] Lustre: Skipped 1 previous similar message [ 5979.617329] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5979.623398] Lustre: Skipped 2 previous similar messages [ 5984.737207] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 5984.742029] Lustre: Skipped 1 previous similar message [ 5985.247862] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5985.341716] Lustre: server umount lustre-MDT0000 complete [ 5987.618331] LustreError: 209189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943851 with bad export cookie 10319485942975326938 [ 5987.620657] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5987.627484] LustreError: 209189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5987.823510] Lustre: server umount lustre-MDT0001 complete [ 5993.226112] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5994.637880] Key type lgssc unregistered [ 5994.800571] LNet: 212842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5994.804183] LNetError: 212842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5994.814349] LNet: Removed LNI 192.168.201.109@tcp [ 5995.220381] Key type .llcrypt unregistered [ 5995.221626] Key type ._llcrypt unregistered [ 6006.808235] Key type ._llcrypt registered [ 6006.812720] Key type .llcrypt registered [ 6006.889942] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6007.581273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6007.654182] alg: No test for adler32 (adler32-zlib) [ 6008.603689] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6008.758980] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6010.376097] Key type lgssc registered [ 6011.120996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6016.002334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6016.013386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.247113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.707945] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6024.988630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.240689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6027.778964] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6031.732485] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6035.204632] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6039.195340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6039.433023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6042.541244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6042.692858] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6046.950137] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6055.911109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 6055.911901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6055.920232] LustreError: Skipped 1 previous similar message [ 6055.937094] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6055.942679] Lustre: Skipped 1 previous similar message [ 6061.027699] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6061.035915] Lustre: Skipped 1 previous similar message [ 6061.783740] Lustre: server umount lustre-OST0000 complete [ 6066.149477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6066.150276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 6066.151724] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6066.161261] Lustre: Skipped 2 previous similar messages [ 6070.205685] Lustre: server umount lustre-MDT0000 complete [ 6071.264458] LustreError: 214825:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6071.284556] LustreError: 214825:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 6072.254770] LustreError: 214068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943936 with bad export cookie 15168098098378485430 [ 6072.255175] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6072.262113] LustreError: 214068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6072.275890] LustreError: 214814:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 6072.279723] LustreError: 214814:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 6072.442169] Lustre: server umount lustre-MDT0001 complete [ 6078.084223] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6079.596274] Key type lgssc unregistered [ 6079.746650] LNet: 217765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6079.750827] LNetError: 217765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6079.766386] LNet: Removed LNI 192.168.201.109@tcp [ 6080.179422] Key type .llcrypt unregistered [ 6080.181869] Key type ._llcrypt unregistered [ 6091.170339] Key type ._llcrypt registered [ 6091.171482] Key type .llcrypt registered [ 6091.234532] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6091.812823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6091.851687] alg: No test for adler32 (adler32-zlib) [ 6092.804612] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6092.943492] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6094.551728] Key type lgssc registered [ 6095.283721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6099.265090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6099.274917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.503432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.362755] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6107.368394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.562653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6109.820047] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6114.238784] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6117.163668] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6120.748590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6121.020152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6124.262872] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6125.115164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6128.754916] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6135.404313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6135.478179] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6135.624737] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6139.127046] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6143.643514] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6144.581776] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6155.808650] Lustre: Permanently deactivating lustre-OST0001 [ 6155.818131] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6161.969932] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6165.712971] Lustre: Permanently reactivating lustre-OST0001 [ 6165.723593] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6165.726027] Lustre: Skipped 2 previous similar messages [ 6170.648135] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6170.663253] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 192.168.201.109@tcp) reconnecting [ 6170.669836] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6170.676997] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.201.109@tcp (at 192.168.201.109@tcp) [ 6170.696111] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6175.712566] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 6175.715872] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6175.723455] LustreError: Skipped 1 previous similar message [ 6175.729407] Lustre: lustre-OST0001: Not available for connect from 192.168.201.109@tcp (stopping) [ 6175.740532] Lustre: Skipped 2 previous similar messages [ 6175.751207] Lustre: Skipped 1 previous similar message [ 6177.249181] Lustre: lustre-OST0001: Not available for connect from 192.168.201.109@tcp (stopping) [ 6177.254203] Lustre: Skipped 1 previous similar message [ 6180.664703] Lustre: server umount lustre-OST0001 complete [ 6182.377296] LustreError: 220825:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.387629] LustreError: 220825:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 6183.397605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 6183.402940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6183.411517] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6186.465469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 6186.468898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6186.478948] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6187.492681] LustreError: 220824:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6187.499226] LustreError: 220824:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 6192.621935] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6192.625518] Lustre: Skipped 2 previous similar messages [ 6192.626867] LustreError: 220826:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6192.636318] LustreError: 220826:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 6197.215130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6197.309554] Lustre: server umount lustre-OST0000 complete [ 6199.777537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 6199.783490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6202.850961] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6202.854686] Lustre: Skipped 1 previous similar message [ 6205.583044] Lustre: server umount lustre-MDT0000 complete [ 6207.774150] LustreError: 218989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944071 with bad export cookie 11888374825839800440 [ 6207.776678] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6207.779426] LustreError: 218989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6207.931513] Lustre: server umount lustre-MDT0001 complete [ 6214.130796] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6215.533646] Key type lgssc unregistered [ 6215.676059] LNet: 224053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6215.678793] LNetError: 224053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6215.689462] LNet: Removed LNI 192.168.201.109@tcp [ 6216.097356] Key type .llcrypt unregistered [ 6216.099061] Key type ._llcrypt unregistered [ 6236.226949] Key type ._llcrypt registered [ 6236.228251] Key type .llcrypt registered [ 6236.274846] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6236.753147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6236.762922] alg: No test for adler32 (adler32-zlib) [ 6237.636943] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6237.750512] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6239.351205] Key type lgssc registered [ 6239.932582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6245.093994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6245.111414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6246.278333] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6246.290127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6246.376919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6248.937679] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6253.671240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.737375] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6253.759918] Lustre: 226234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6253.847593] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.109@tcp (not set up) [ 6253.936058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6256.307228] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6260.600494] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6263.845995] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6267.970038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6268.121968] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6268.253546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6271.294659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6271.442908] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6275.251650] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6278.103270] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6285.036827] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6293.203596] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6297.987155] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6307.898567] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6329.321556] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6329.324273] Lustre: Skipped 2 previous similar messages [ 6365.207682] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6365.210759] Lustre: Skipped 3 previous similar messages [ 6432.918048] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6432.924943] Lustre: Skipped 8 previous similar messages [ 6445.540596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 6445.543362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6445.545403] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6445.553993] Lustre: Skipped 1 previous similar message [ 6448.156899] Lustre: server umount lustre-OST0000 complete [ 6450.657452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6450.666016] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6450.669352] Lustre: Skipped 2 previous similar messages [ 6455.776574] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6455.780148] Lustre: Skipped 1 previous similar message [ 6455.967429] Lustre: server umount lustre-MDT0000 complete [ 6457.332326] LustreError: 228803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944321 with bad export cookie 19859025487160843 [ 6457.335511] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6457.336559] LustreError: 228803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6457.452927] Lustre: server umount lustre-MDT0001 complete [ 6461.517185] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6462.724878] Key type lgssc unregistered [ 6462.856358] LNet: 231058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6462.860647] LNetError: 231058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6462.871497] LNet: Removed LNI 192.168.201.109@tcp [ 6463.224873] Key type .llcrypt unregistered [ 6463.227047] Key type ._llcrypt unregistered [ 6473.738140] Key type ._llcrypt registered [ 6473.739148] Key type .llcrypt registered [ 6473.783762] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6474.209038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6474.238391] alg: No test for adler32 (adler32-zlib) [ 6475.110484] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6475.205670] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6476.807162] Key type lgssc registered [ 6477.244420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6480.510520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6480.516522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.697937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6483.566314] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6487.530678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6487.707828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6489.471168] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6493.752146] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6496.693117] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6499.954163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6500.165335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6503.115871] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6506.358477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6507.015805] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6521.824411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 6521.827221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6521.833546] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6522.852716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.109@tcp failed: rc = -107 [ 6522.854490] Lustre: lustre-OST0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6522.859196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6522.869025] Lustre: Skipped 1 previous similar message [ 6525.048845] Lustre: server umount lustre-OST0000 complete [ 6527.969299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.109@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6527.970499] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6527.977147] Lustre: Skipped 1 previous similar message [ 6533.090503] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6533.103205] Lustre: Skipped 2 previous similar messages [ 6538.209806] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.109@tcp (stopping) [ 6538.212268] Lustre: Skipped 1 previous similar message [ 6541.279172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6541.360798] Lustre: server umount lustre-MDT0000 complete [ 6542.304456] LustreError: 232446:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.109@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6542.310605] LustreError: 232446:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 6543.027343] LustreError: 232427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944407 with bad export cookie 14549312454482905100 [ 6543.027809] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6543.033710] LustreError: 232427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6543.162244] Lustre: server umount lustre-MDT0001 complete [ 6548.028438] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6549.543657] Key type lgssc unregistered [ 6549.735484] LNet: 236131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6549.746623] LNetError: 236131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6549.765644] LNet: Removed LNI 192.168.201.109@tcp [ 6550.232283] Key type .llcrypt unregistered [ 6550.235099] Key type ._llcrypt unregistered [ 6561.911424] Key type ._llcrypt registered [ 6561.912625] Key type .llcrypt registered [ 6561.958269] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6562.409255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6562.439785] alg: No test for adler32 (adler32-zlib) [ 6563.348789] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6563.476245] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6565.071187] Key type lgssc registered [ 6565.622788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6830.985787] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6832.284523] Key type lgssc unregistered [ 6832.436641] LNet: 238132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6832.443102] LNetError: 238132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6832.462477] LNet: Removed LNI 192.168.201.109@tcp [ 6832.814658] Key type .llcrypt unregistered [ 6832.816812] Key type ._llcrypt unregistered [ 6842.225112] Key type ._llcrypt registered [ 6842.226281] Key type .llcrypt registered [ 6842.271183] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 6849.188949] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6849.619217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6849.633786] alg: No test for adler32 (adler32-zlib) [ 6850.499520] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6850.607599] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6852.191104] Key type lgssc registered [ 6852.589537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6864.514351] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6601 sec ========== 21:58:47 (1784944727) [ 6865.264879] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:58:48 (1784944728) === [ 6866.775714] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:58:50 (1784944730) === [ 6883.820362] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6885.097908] Key type lgssc unregistered [ 6885.267681] LNet: 243336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6885.271471] LNetError: 243336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6885.284665] LNet: Removed LNI 192.168.201.109@tcp [ 6885.653306] Key type .llcrypt unregistered [ 6885.655031] Key type ._llcrypt unregistered