[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 472289468 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003107] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010137] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013066] *** VALIDATE selinux *** [ 0.020600] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024679] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026092] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027104] *** VALIDATE tmpfs *** [ 0.028476] *** VALIDATE proc *** [ 0.029200] *** VALIDATE cgroup *** [ 0.030009] *** VALIDATE cgroup2 *** [ 0.031213] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036032] Spectre V2 : User space: Vulnerable [ 0.037012] Speculative Store Bypass: Vulnerable [ 0.040296] debug: unmapping init [mem 0xffffffff8a259000-0xffffffff8a260fff] [ 0.042197] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043722] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044026] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048016] ... max period: 00007fffffffffff [ 0.049015] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051301] rcu: Hierarchical SRCU implementation. [ 0.053466] smp: Bringing up secondary CPUs ... [ 0.054591] x86: Booting SMP configuration: [ 0.055030] .... node #0, CPUs: #1 #2 #3 [ 0.058369] smp: Brought up 1 node, 4 CPUs [ 0.060016] smpboot: Max logical packages: 1 [ 0.061015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126806] node 0 deferred pages initialised in 62ms [ 0.130310] devtmpfs: initialized [ 0.131197] x86/mm: Memory block size: 128MB [ 0.133481] gcov: version magic: 0x41383552 [ 0.136291] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.137066] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.138000] pinctrl core: initialized pinctrl subsystem [ 0.139126] [ 0.139499] ************************************************************* [ 0.141011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143009] ** ** [ 0.145012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147011] ** ** [ 0.149011] ** This means that this kernel is built to expose internal ** [ 0.151012] ** IOMMU data structures, which may compromise security on ** [ 0.153010] ** your system. ** [ 0.155011] ** ** [ 0.156009] ** If you see this message and you are not debugging the ** [ 0.158012] ** kernel, report this immediately to your vendor! ** [ 0.160016] ** ** [ 0.163016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164008] ************************************************************* [ 0.166749] NET: Registered protocol family 16 [ 0.168419] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175075] cpuidle: using governor menu [ 0.175858] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177444] PCI: Using configuration type 1 for base access [ 0.179129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185135] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188101] cryptd: max_cpu_qlen set to 1000 [ 0.191297] ACPI: Added _OSI(Module Device) [ 0.193018] ACPI: Added _OSI(Processor Device) [ 0.194014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200508] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207424] ACPI: Interpreter enabled [ 0.208063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210013] ACPI: Using IOAPIC for interrupt routing [ 0.212108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214469] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222852] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.225050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.230079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.234625] acpiphp: Slot [2] registered [ 0.236183] acpiphp: Slot [5] registered [ 0.238171] acpiphp: Slot [6] registered [ 0.239145] acpiphp: Slot [7] registered [ 0.241110] acpiphp: Slot [8] registered [ 0.243183] acpiphp: Slot [9] registered [ 0.244170] acpiphp: Slot [10] registered [ 0.246113] acpiphp: Slot [3] registered [ 0.247109] acpiphp: Slot [4] registered [ 0.249145] acpiphp: Slot [11] registered [ 0.251124] acpiphp: Slot [12] registered [ 0.252098] acpiphp: Slot [13] registered [ 0.254104] acpiphp: Slot [14] registered [ 0.256097] acpiphp: Slot [15] registered [ 0.257105] acpiphp: Slot [16] registered [ 0.258081] acpiphp: Slot [17] registered [ 0.258936] acpiphp: Slot [18] registered [ 0.260123] acpiphp: Slot [19] registered [ 0.261106] acpiphp: Slot [20] registered [ 0.262103] acpiphp: Slot [21] registered [ 0.263020] acpiphp: Slot [22] registered [ 0.263868] acpiphp: Slot [23] registered [ 0.265125] acpiphp: Slot [24] registered [ 0.267069] acpiphp: Slot [25] registered [ 0.268027] acpiphp: Slot [26] registered [ 0.269088] acpiphp: Slot [27] registered [ 0.270041] acpiphp: Slot [28] registered [ 0.270954] acpiphp: Slot [29] registered [ 0.272131] acpiphp: Slot [30] registered [ 0.273090] acpiphp: Slot [31] registered [ 0.274061] PCI host bridge to bus 0000:00 [ 0.275020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.277025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.279026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.280023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.282023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.284027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.285189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.288125] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.290388] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.301773] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.306497] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.308019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.310011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.311011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.313459] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.315792] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.318048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.321891] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.326879] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.338018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.341018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.345965] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.358023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.367019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.385022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.395115] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.400011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.405013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.417021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.427000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.434014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.438014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.450018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.459601] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.469023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.477023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.499017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.510000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.520015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.527017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.551017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.558260] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.565018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.571019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.588024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.596581] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.598357] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.599250] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.600323] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.602144] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.604200] iommu: Default domain type: Passthrough [ 0.605309] SCSI subsystem initialized [ 0.606089] ACPI: bus type USB registered [ 0.606994] usbcore: registered new interface driver usbfs [ 0.608037] usbcore: registered new interface driver hub [ 0.609049] usbcore: registered new device driver usb [ 0.610104] pps_core: LinuxPPS API ver. 1 registered [ 0.611011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.614052] PTP clock support registered [ 0.615121] EDAC MC: Ver: 3.0.0 [ 0.617135] PCI: Using ACPI for IRQ routing [ 0.618894] NetLabel: Initializing [ 0.619009] NetLabel: domain hash size = 128 [ 0.620009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.621096] NetLabel: unlabeled traffic allowed by default [ 0.623098] vgaarb: loaded [ 0.624133] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.625125] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.631242] clocksource: Switched to clocksource kvm-clock [ 0.735694] VFS: Disk quotas dquot_6.6.0 [ 0.738400] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.740192] *** VALIDATE ramfs *** [ 0.741111] *** VALIDATE hugetlbfs *** [ 0.742255] pnp: PnP ACPI init [ 0.744753] pnp: PnP ACPI: found 6 devices [ 0.762647] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.766172] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.768397] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.769885] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.771392] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.773635] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.776603] NET: Registered protocol family 2 [ 0.779402] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.785087] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.789146] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.794806] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.798669] TCP: Hash tables configured (established 65536 bind 65536) [ 0.801638] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.804534] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.807668] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.811084] NET: Registered protocol family 1 [ 0.815079] RPC: Registered named UNIX socket transport module. [ 0.817445] RPC: Registered udp transport module. [ 0.819048] RPC: Registered tcp transport module. [ 0.820629] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.822885] NET: Registered protocol family 44 [ 0.824232] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.826663] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.828681] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.830553] PCI: CLS 0 bytes, default 64 [ 0.831848] Unpacking initramfs... [ 2.239652] debug: unmapping init [mem 0xffff8d51fcc54000-0xffff8d51fffbffff] [ 2.243429] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.245744] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.248664] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.789348] Initialise system trusted keyrings [ 2.790860] Key type blacklist registered [ 2.793204] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.801151] zbud: loaded [ 2.803677] *** VALIDATE nfs *** [ 2.804970] *** VALIDATE nfs4 *** [ 2.806450] pstore: using deflate compression [ 2.810924] Platform Keyring initialized [ 2.920669] NET: Registered protocol family 38 [ 2.922873] Key type asymmetric registered [ 2.924549] Asymmetric key parser 'x509' registered [ 2.926321] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.928619] io scheduler mq-deadline registered [ 2.930274] io scheduler kyber registered [ 2.932146] io scheduler bfq registered [ 2.933892] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.937355] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.939703] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.942639] ACPI: Power Button [PWRF] [ 2.948323] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.956713] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.972730] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.982848] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.003168] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.031601] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.060393] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.066606] Non-volatile memory driver v1.3 [ 3.068135] Linux agpgart interface v0.103 [ 3.102954] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.105755] vda: detected capacity change from 0 to 75071488 [ 3.125600] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.128523] vdb: detected capacity change from 0 to 1073741824 [ 3.145550] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.148025] vdc: detected capacity change from 0 to 2621440000 [ 3.162060] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.164752] vdd: detected capacity change from 0 to 2621440000 [ 3.177661] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.180258] vde: detected capacity change from 0 to 4294967296 [ 3.194612] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.198039] vdf: detected capacity change from 0 to 4294967296 [ 3.209305] libphy: Fixed MDIO Bus: probed [ 3.215951] usbcore: registered new interface driver usbserial_generic [ 3.218906] usbserial: USB Serial support registered for generic [ 3.221570] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.225537] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.227273] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.229833] mousedev: PS/2 mouse device common for all mice [ 3.233529] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.236758] rtc_cmos 00:05: RTC can wake from S4 [ 3.240459] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.240598] rtc_cmos 00:05: registered as rtc0 [ 3.245414] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.247873] intel_pstate: CPU model not supported [ 3.248189] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.253527] hid: raw HID events driver (C) Jiri Kosina [ 3.256135] usbcore: registered new interface driver usbhid [ 3.258406] usbhid: USB HID core driver [ 3.259997] drop_monitor: Initializing network drop monitor service [ 3.262516] Initializing XFRM netlink socket [ 3.264187] NET: Registered protocol family 10 [ 3.268427] Segment Routing with IPv6 [ 3.269723] NET: Registered protocol family 17 [ 3.271527] mpls_gso: MPLS GSO support [ 3.277687] RAS: Correctable Errors collector initialized. [ 3.280157] AVX version of gcm_enc/dec engaged. [ 3.281752] AES CTR mode by8 optimization enabled [ 3.353866] sched_clock: Marking stable (3353842712, 0)->(4215399470, -861556758) [ 3.356532] registered taskstats version 1 [ 3.358733] Loading compiled-in X.509 certificates [ 3.360233] zswap: loaded using pool lzo/zbud [ 3.380764] Key type big_key registered [ 3.392085] Key type encrypted registered [ 3.393631] ima: No TPM chip found, activating TPM-bypass! [ 3.395202] ima: Allocated hash algorithm: sha1 [ 3.396976] ima: No architecture policies found [ 3.398543] evm: Initialising EVM extended attributes: [ 3.399771] evm: security.selinux [ 3.400891] evm: security.ima [ 3.401976] evm: security.capability [ 3.403056] evm: HMAC attrs: 0x1 [ 3.405275] rtc_cmos 00:05: setting system clock to 2026-08-26 21:56:24 UTC (1787781384) [ 3.412283] debug: unmapping init [mem 0xffffffff8b203000-0xffffffff8b3fffff] [ 3.415201] debug: unmapping init [mem 0xffffffff89f82000-0xffffffff8a258fff] [ 3.420108] Write protecting the kernel read-only data: 28672k [ 3.423435] debug: unmapping init [mem 0xffffffff88603000-0xffffffff887fffff] [ 3.425668] debug: unmapping init [mem 0xffffffff88f14000-0xffffffff88ffffff] [ 3.459423] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.467546] systemd[1]: Detected virtualization kvm. [ 3.469313] systemd[1]: Detected architecture x86-64. [ 3.471129] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.499640] systemd[1]: No hostname configured. [ 3.501256] systemd[1]: Set hostname to . [ 3.503374] random: systemd: uninitialized urandom read (16 bytes read) [ 3.506027] systemd[1]: Initializing machine ID from random generator. [ 3.655840] random: systemd: uninitialized urandom read (16 bytes read) [ 3.658922] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.663616] random: systemd: uninitialized urandom read (16 bytes read) [ 3.665951] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.670196] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.293091] device-mapper: uevent: version 1.0.3 [ 4.294570] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.007613] virtio_net virtio0 ens2: renamed from eth0 [ 5.014646] scsi host0: ata_piix [ 5.039410] random: fast init done [ 5.054465] scsi host1: ata_piix [ 5.163783] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.166056] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.546344] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.907870] random: crng init done [ 9.910991] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.381475] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.538583] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.766490] SELinux: Disabled at runtime. [ 11.828760] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.835557] systemd[1]: Detected virtualization kvm. [ 11.837237] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.315580] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.323035] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.328247] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.332268] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.335389] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.343607] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.347604] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details.[ 12.519697] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.876593] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.229314] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.248409] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.400372] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.409422] EDAC sbridge: Ver: 1.1.2 [ 15.013659] Key type dns_resolver registered [ 15.314295] NFS: Registering the id_resolver key type [ 15.316177] Key type id_resolver registered [ 15.317502] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg103-server login: [ 49.902279] libcfs: loading out-of-tree module taints kernel. [ 50.018592] Key type ._llcrypt registered [ 50.023530] Key type .llcrypt registered [ 50.153797] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 73.770624] hrtimer: interrupt took 11921430 ns [ 74.704495] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 76.423829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 76.498895] alg: No test for adler32 (adler32-zlib) [ 77.868416] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 79.203173] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 81.143161] Key type lgssc registered [ 84.175179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.523604] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 162.512892] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 176.566875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 176.651464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.030298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 178.077068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.225351] Lustre: lustre-MDT0000: new disk, initializing [ 178.358083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.386229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 183.026864] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.441584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.578566] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 198.655726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 198.665633] Lustre: Skipped 1 previous similar message [ 198.838094] Lustre: lustre-MDT0001: new disk, initializing [ 198.945671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.979501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 198.996725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.596508] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.519651] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 219.993761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.364143] Lustre: lustre-OST0000: new disk, initializing [ 220.371555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.375972] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.409596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.379075] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.948790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.959680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.077877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.826629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.024666] Lustre: lustre-OST0001: new disk, initializing [ 242.030750] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 242.048896] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.187470] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 248.722637] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.451781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.478363] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 250.574871] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 263.257114] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.303474] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.535855] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.344724] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing yml_node [ 287.156364] Lustre: DEBUG MARKER: Client: 2.17.57.56 [ 289.876469] Lustre: DEBUG MARKER: MDS: 2.17.57.56 [ 293.145129] Lustre: DEBUG MARKER: OSS: 2.17.57.56 [ 294.841231] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 18:01:15 EDT 2026 [ 314.646484] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 316.735765] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 337.382538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 337.393869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.412512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.745753] Lustre: server umount lustre-MDT0000 complete [ 344.454342] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787781725 with bad export cookie 16654887946344489197 [ 344.456105] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 344.466575] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 348.162395] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.162431] LustreError: 9472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.174017] Lustre: Skipped 4 previous similar messages [ 348.174257] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.174263] Lustre: Skipped 3 previous similar messages [ 348.235565] LustreError: 9472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 349.155161] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.184060] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 350.899734] Lustre: server umount lustre-MDT0001 complete [ 354.811580] Lustre: server umount lustre-OST0000 complete [ 358.986950] Lustre: server umount lustre-OST0001 complete [ 366.970466] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 379.067801] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 416.314668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.637286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 416.663672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.748464] Lustre: lustre-MDT0000: new disk, initializing [ 416.851903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.895538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 422.095420] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 432.601932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.783285] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.907989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 432.917887] Lustre: Skipped 1 previous similar message [ 433.037346] Lustre: lustre-MDT0001: new disk, initializing [ 433.142322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 433.184531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 433.199806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.092570] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 449.814894] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 456.520780] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 464.129882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 464.443087] Lustre: lustre-OST0000: new disk, initializing [ 464.447360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 464.458208] Lustre: 17060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 464.574638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.304214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 466.322196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 466.448550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.038460] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 481.991832] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 487.930656] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 488.195167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 494.043194] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 494.434286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 497.123540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 497.124807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.128606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 497.128620] Lustre: Skipped 3 previous similar messages [ 497.145340] LustreError: Skipped 1 previous similar message [ 497.169880] Lustre: Skipped 1 previous similar message [ 502.255854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.261205] Lustre: Skipped 2 previous similar messages [ 503.265457] Lustre: server umount lustre-OST0000 complete [ 509.935034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 509.940047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.950163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 513.988610] Lustre: server umount lustre-MDT0000 complete [ 517.601295] LustreError: 15101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.616605] LustreError: 15101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 518.964668] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787781900 with bad export cookie 16654887946344490730 [ 518.970443] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.976607] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 519.451951] Lustre: server umount lustre-MDT0001 complete [ 524.266806] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 18:05:04 (1787781904) [ 533.152250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.824461] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.002325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 539.105231] LustreError: 19326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.789363] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 548.326671] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.347643] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 550.804540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.975478] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 565.132583] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 574.178350] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 582.772143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 590.207753] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 597.964735] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 611.809920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 611.817858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.832556] Lustre: Skipped 1 previous similar message [ 611.843533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 611.851126] Lustre: Skipped 2 previous similar messages [ 616.691319] Lustre: server umount lustre-OST0000 complete [ 623.072401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 623.083762] LustreError: Skipped 1 previous similar message [ 623.090469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.099688] Lustre: Skipped 1 previous similar message [ 627.476502] Lustre: server umount lustre-MDT0000 complete [ 629.218829] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.239868] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 631.588587] LustreError: 19306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782012 with bad export cookie 16654887946344491612 [ 631.591866] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 631.604786] LustreError: 19306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 631.966791] Lustre: server umount lustre-MDT0001 complete [ 645.798392] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 649.554543] Key type lgssc unregistered [ 649.820616] LNet: 22880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.829848] LNetError: 22880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.841842] LNet: Removed LNI 192.168.201.103@tcp [ 650.685216] Key type .llcrypt unregistered [ 650.689359] Key type ._llcrypt unregistered [ 674.804678] Key type ._llcrypt registered [ 674.806928] Key type .llcrypt registered [ 674.894949] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 676.562614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 676.638369] alg: No test for adler32 (adler32-zlib) [ 677.704228] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 678.008229] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 679.743300] Key type lgssc registered [ 680.846848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 688.749726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 688.778760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.270833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 695.402550] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 700.400290] LustreError: 24295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.413357] LustreError: 24295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 703.909581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.255497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 709.193504] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 716.339841] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 722.347172] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 730.014632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 730.521767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 735.685171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 737.163552] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 746.260794] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 766.434335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 766.444351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.475292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 769.507211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 769.512195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 769.526444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.570874] Lustre: Skipped 1 previous similar message [ 770.843636] Lustre: server umount lustre-OST0000 complete [ 776.166252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 776.173743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.202167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.748028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.750837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 782.025680] Lustre: server umount lustre-MDT0000 complete [ 784.865027] LustreError: 24292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 784.889654] LustreError: 24292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 785.828300] LustreError: 24277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782166 with bad export cookie 15747408800191249877 [ 785.850764] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 786.159185] Lustre: server umount lustre-MDT0001 complete [ 799.004683] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 801.992860] Key type lgssc unregistered [ 802.303583] LNet: 28274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.316226] LNetError: 28274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.331503] LNet: Removed LNI 192.168.201.103@tcp [ 803.367793] Key type .llcrypt unregistered [ 803.370398] Key type ._llcrypt unregistered [ 829.651362] Key type ._llcrypt registered [ 829.668216] Key type .llcrypt registered [ 829.871820] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 831.602853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 831.682329] alg: No test for adler32 (adler32-zlib) [ 832.714281] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 832.990083] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 834.655251] Key type lgssc registered [ 836.414273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 845.781852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 845.820544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.455243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 853.356772] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 857.575136] LustreError: 29689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 857.592584] LustreError: 29689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 862.688607] LustreError: 29690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 863.998125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 864.460159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 869.060775] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 877.003929] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 883.526380] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 898.061765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 898.449400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.338812] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 912.513720] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 917.879866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 925.161535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 925.174773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.206562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 926.023918] Lustre: server umount lustre-OST0000 complete [ 931.297662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 931.308886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.339074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 933.349839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.360156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 933.371578] Lustre: Skipped 1 previous similar message [ 938.486522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 938.506825] Lustre: Skipped 1 previous similar message [ 940.708215] Lustre: server umount lustre-MDT0000 complete [ 943.585458] LustreError: 29686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 943.610035] LustreError: 29686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 944.673155] LustreError: 29671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782325 with bad export cookie 8500794358206944104 [ 944.682748] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 944.687039] LustreError: 29671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 944.922831] Lustre: server umount lustre-MDT0001 complete [ 954.545934] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 957.637711] Key type lgssc unregistered [ 957.872934] LNet: 33572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.878337] LNetError: 33572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.895993] LNet: Removed LNI 192.168.201.103@tcp [ 958.804771] Key type .llcrypt unregistered [ 958.807205] Key type ._llcrypt unregistered [ 985.950085] Key type ._llcrypt registered [ 985.951769] Key type .llcrypt registered [ 986.160696] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 987.682466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 987.705988] alg: No test for adler32 (adler32-zlib) [ 988.908856] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 989.303804] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 991.095267] Key type lgssc registered [ 992.520944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 999.307242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 999.356796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.003171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.873730] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1011.173871] LustreError: 34984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1011.189414] LustreError: 34984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1013.459787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.770637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1018.344768] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1025.733827] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1031.698529] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1039.043576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1039.413816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.595760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1047.015607] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1055.865864] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1072.103950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1072.113063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.134517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1074.152586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1074.161406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1074.169437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.663850] Lustre: server umount lustre-OST0000 complete [ 1080.296877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1080.306649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.327918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1080.339903] Lustre: Skipped 1 previous similar message [ 1084.387740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.389414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.433243] Lustre: Skipped 1 previous similar message [ 1086.367769] Lustre: server umount lustre-MDT0000 complete [ 1089.505688] LustreError: 34984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.545621] LustreError: 34984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1090.320199] LustreError: 34965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782471 with bad export cookie 8543378527959713669 [ 1090.331717] LustreError: 34965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1090.333552] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1090.680114] Lustre: server umount lustre-MDT0001 complete [ 1102.942532] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1105.932202] Key type lgssc unregistered [ 1106.317716] LNet: 38547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.326661] LNetError: 38547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1107.379337] LNet: Removed LNI 192.168.201.103@tcp [ 1108.523559] Key type .llcrypt unregistered [ 1108.528927] Key type ._llcrypt unregistered [ 1136.426540] Key type ._llcrypt registered [ 1136.437514] Key type .llcrypt registered [ 1136.578350] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1137.890846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1137.994951] alg: No test for adler32 (adler32-zlib) [ 1139.079421] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 1139.291823] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1141.025082] Key type lgssc registered [ 1142.625315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1150.509417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1150.524523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.960807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.308100] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1162.212759] LustreError: 39965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.237625] LustreError: 39965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1164.907144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.306646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1170.370451] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1179.316415] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1185.064553] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1191.402862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1191.672885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1197.494844] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1197.804460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1204.797442] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1215.459318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.462716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.523859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1216.846823] Lustre: lustre-OST0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1216.853514] Lustre: Skipped 1 previous similar message [ 1219.964885] Lustre: server umount lustre-OST0000 complete [ 1240.032624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.037566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1240.045799] Lustre: Skipped 2 previous similar messages [ 1245.154337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1245.157822] Lustre: Skipped 2 previous similar messages [ 1245.901724] Lustre: server umount lustre-MDT0000 complete [ 1249.446957] LustreError: 39946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782630 with bad export cookie 1334123931267268276 [ 1249.458551] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1249.464971] LustreError: 39946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1255.865718] Lustre: server umount lustre-MDT0001 complete [ 1267.890553] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1270.782406] Key type lgssc unregistered [ 1271.100891] LNet: 43680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1271.107932] LNetError: 43680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.166224] LNet: Removed LNI 192.168.201.103@tcp [ 1272.956477] Key type .llcrypt unregistered [ 1272.958890] Key type ._llcrypt unregistered [ 1301.483882] Key type ._llcrypt registered [ 1301.485848] Key type .llcrypt registered [ 1301.577228] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1302.789614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1302.837996] alg: No test for adler32 (adler32-zlib) [ 1303.901609] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 1304.117733] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1305.767267] Key type lgssc registered [ 1307.022897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1315.684803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1315.698328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.202470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.735584] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1327.587525] LustreError: 45096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.628322] LustreError: 45096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1331.721061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.893962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1336.395456] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1347.269514] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1355.008501] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1364.863535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1365.609926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.619242] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1381.550721] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1387.136290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1392.468970] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1392.610683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.627026] Lustre: Skipped 2 previous similar messages [ 1394.068936] Lustre: server umount lustre-MDT0000 complete [ 1397.568098] LustreError: 45092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1397.624265] LustreError: 45077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782778 with bad export cookie 1981093112728711583 [ 1397.646091] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1397.648269] LustreError: 45077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1397.736528] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.766578] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1397.782653] Lustre: Skipped 3 previous similar messages [ 1402.675704] LustreError: 45091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.709639] LustreError: 45091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1402.718669] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 1404.414252] Lustre: server umount lustre-MDT0001 complete [ 1426.793869] Lustre: server umount lustre-OST0000 complete [ 1447.092321] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1449.029569] Key type lgssc unregistered [ 1449.514563] LNet: 48962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.527745] LNetError: 48962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.554575] LNet: Removed LNI 192.168.201.103@tcp [ 1450.562826] Key type .llcrypt unregistered [ 1450.570469] Key type ._llcrypt unregistered [ 1478.464941] Key type ._llcrypt registered [ 1478.468876] Key type .llcrypt registered [ 1478.667387] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1480.780704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1480.867281] alg: No test for adler32 (adler32-zlib) [ 1481.998334] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 1482.387181] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1484.087212] Key type lgssc registered [ 1485.778822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1495.231839] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1495.242847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1510.753084] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1515.871496] LustreError: 50352:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 1542.623924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.305521] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1556.828659] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1722.407760] Lustre: server umount lustre-OST0000 complete [ 1741.011495] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1744.366938] Key type lgssc unregistered [ 1744.653173] LNet: 52032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.661693] LNetError: 52032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1744.683350] LNet: Removed LNI 192.168.201.103@tcp [ 1745.554238] Key type .llcrypt unregistered [ 1745.562670] Key type ._llcrypt unregistered [ 1770.077375] Key type ._llcrypt registered [ 1770.080856] Key type .llcrypt registered [ 1770.182409] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1771.715183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1771.749253] alg: No test for adler32 (adler32-zlib) [ 1772.755790] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 1772.983396] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1774.680524] Key type lgssc registered [ 1775.865799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1784.050794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1784.087693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.619067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.129650] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1795.049325] LustreError: 53448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1795.072962] LustreError: 53448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1798.746168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.266343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1804.453124] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1812.345651] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1818.663874] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1826.293269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1826.855681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.939197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1834.595928] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1843.859757] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1849.320967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.322878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1849.327240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.354688] Lustre: Skipped 1 previous similar message [ 1853.233816] Lustre: server umount lustre-OST0000 complete [ 1859.552855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.559548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1859.579069] Lustre: Skipped 1 previous similar message [ 1859.593348] Lustre: Skipped 2 previous similar messages [ 1864.678204] LustreError: 54176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1864.703794] LustreError: 54176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1864.818914] Lustre: server umount lustre-MDT0000 complete [ 1869.792975] LustreError: 53448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.838608] LustreError: 53448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1870.026772] LustreError: 53430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783251 with bad export cookie 8813367424081658907 [ 1870.045681] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1870.055276] LustreError: 53430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1870.620140] Lustre: server umount lustre-MDT0001 complete [ 1881.818756] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1885.229985] Key type lgssc unregistered [ 1885.745975] LNet: 56987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.760987] LNetError: 56987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.780809] LNet: Removed LNI 192.168.201.103@tcp [ 1886.808750] Key type .llcrypt unregistered [ 1886.822430] Key type ._llcrypt unregistered [ 1914.652398] Key type ._llcrypt registered [ 1914.654497] Key type .llcrypt registered [ 1914.737515] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1915.696050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1915.781046] alg: No test for adler32 (adler32-zlib) [ 1916.884469] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 1917.095727] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1918.748340] Key type lgssc registered [ 1920.121320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1928.707943] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1928.743161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1944.288321] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1949.408558] LustreError: 58375:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 1976.095361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.437216] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1994.091650] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2001.874969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.400682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.480652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2007.532038] LustreError: 58401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.598232] LustreError: 58401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2008.628169] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2010.614564] LustreError: 59412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.632895] LustreError: 59412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2015.722514] LustreError: 59413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.766903] LustreError: 59413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2018.888748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.520656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2025.320094] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2032.913875] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2039.322706] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2043.361212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2043.368327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.386204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2044.901086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2044.907093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.941534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2047.936904] Lustre: server umount lustre-OST0000 complete [ 2070.498284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2070.506217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.537463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2070.551279] Lustre: Skipped 2 previous similar messages [ 2073.058920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.062306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2073.085764] Lustre: Skipped 1 previous similar message [ 2076.614774] Lustre: server umount lustre-MDT0000 complete [ 2079.200509] LustreError: 59417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.223437] LustreError: 59417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2080.755776] LustreError: 58382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783461 with bad export cookie 5243035638167939365 [ 2080.763622] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2080.773905] LustreError: 58382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2087.238061] Lustre: server umount lustre-MDT0001 complete [ 2099.170247] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2101.843583] Key type lgssc unregistered [ 2102.182974] LNet: 62124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.187631] LNetError: 62124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2103.205855] LNet: Removed LNI 192.168.201.103@tcp [ 2104.482220] Key type .llcrypt unregistered [ 2104.484025] Key type ._llcrypt unregistered [ 2130.515913] Key type ._llcrypt registered [ 2130.518784] Key type .llcrypt registered [ 2130.624334] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2132.124912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2132.384699] alg: No test for adler32 (adler32-zlib) [ 2133.539752] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 2133.855798] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2135.687180] Key type lgssc registered [ 2137.079343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2146.436556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2146.473571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.030575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.843079] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2158.569807] LustreError: 63544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2158.583892] LustreError: 63544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2163.002137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.436442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2167.911615] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2174.460671] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2180.179325] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2186.641144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2186.947134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.923905] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2194.115848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2200.275623] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2225.120139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2225.140397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.165850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2225.179810] Lustre: Skipped 1 previous similar message [ 2226.624972] Lustre: server umount lustre-OST0000 complete [ 2235.361905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.377480] Lustre: Skipped 1 previous similar message [ 2235.385581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2235.401814] Lustre: Skipped 1 previous similar message [ 2236.849418] Lustre: server umount lustre-MDT0000 complete [ 2240.480881] LustreError: 64273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.501228] LustreError: 64273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2241.861369] LustreError: 63527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783622 with bad export cookie 12534775404691503195 [ 2241.877639] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2241.883126] LustreError: 63527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2242.369403] Lustre: server umount lustre-MDT0001 complete [ 2254.072961] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2257.559851] Key type lgssc unregistered [ 2257.923111] LNet: 67100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2257.926556] LNetError: 67100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2257.944632] LNet: Removed LNI 192.168.201.103@tcp [ 2259.130139] Key type .llcrypt unregistered [ 2259.135351] Key type ._llcrypt unregistered [ 2286.461707] Key type ._llcrypt registered [ 2286.466287] Key type .llcrypt registered [ 2286.565881] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2287.582211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2287.697942] alg: No test for adler32 (adler32-zlib) [ 2288.726889] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 2289.019357] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2290.687219] Key type lgssc registered [ 2291.793782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2299.501862] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 18:34:40 (1787783680) [ 2307.240234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2307.265317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.712755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2313.370507] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2318.306310] LustreError: 68803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.330662] LustreError: 68803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2323.424304] LustreError: 68804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.838396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.268062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2330.616740] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2339.048650] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2344.757479] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2350.968056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2351.377756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2353.464197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2357.142912] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2364.818670] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2373.578540] Lustre: Failing over lustre-MDT0000 [ 2373.832376] Lustre: server umount lustre-MDT0000 complete [ 2376.165443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.172685] LustreError: 69531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.176882] Lustre: Skipped 2 previous similar messages [ 2376.189761] LustreError: 69531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2381.282670] LustreError: 68800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.303702] LustreError: 68800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2383.538983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.656123] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.749723] Lustre: *** cfs_fail_loc=135, val=0*** [ 2383.811756] LustreError: 71892:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2383.816594] LustreError: 71892:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.103@tcp: cfg command failed: rc = -2 [ 2383.822892] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2383.822892] [ 2383.829855] LustreError: MGC192.168.201.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2383.843729] LustreError: 71880:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2383.850908] LustreError: 71880:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2383.859116] LustreError: 71880:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2383.865805] LustreError: 71880:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2383.897961] Lustre: server umount lustre-MDT0000 complete [ 2383.908344] LustreError: 71880:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2386.401928] LustreError: 68798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.424070] LustreError: 68798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2389.887607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.056231] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2390.407879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.474885] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2391.861602] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2395.349110] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2395.633656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2395.685323] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2395.750682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2405.876947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.912684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2405.920617] Lustre: Skipped 1 previous similar message [ 2408.486757] Lustre: server umount lustre-OST0000 complete [ 2416.099984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2416.100398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.123307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2416.134114] Lustre: Skipped 2 previous similar messages [ 2416.148028] Lustre: Skipped 1 previous similar message [ 2421.220583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.235874] Lustre: Skipped 1 previous similar message [ 2426.335193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2426.339069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2426.350066] Lustre: Skipped 1 previous similar message [ 2426.614630] Lustre: server umount lustre-MDT0000 complete [ 2430.450174] LustreError: 68784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783811 with bad export cookie 10409294576076755904 [ 2430.466844] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.827568] Lustre: server umount lustre-MDT0001 complete [ 2441.870315] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2444.746569] Key type lgssc unregistered [ 2445.045808] LNet: 73675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2445.073436] LNetError: 73675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2445.097079] LNet: Removed LNI 192.168.201.103@tcp [ 2445.962563] Key type .llcrypt unregistered [ 2445.964279] Key type ._llcrypt unregistered [ 2467.557269] Key type ._llcrypt registered [ 2467.559580] Key type .llcrypt registered [ 2467.670807] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2468.780748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2468.835183] alg: No test for adler32 (adler32-zlib) [ 2469.917899] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 2470.180738] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2471.855184] Key type lgssc registered [ 2472.963362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2479.985873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2480.001870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.498329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2486.868897] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2491.874107] LustreError: 75069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.894297] LustreError: 75069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2494.882842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.308790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2500.089606] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2507.550188] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2512.739982] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2518.509848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.745898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.041488] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2525.962770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2530.368365] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2535.519504] Lustre: Failing over lustre-MDT0000 [ 2535.753391] Lustre: server umount lustre-MDT0000 complete [ 2536.423313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.431251] LustreError: 77560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.446683] Lustre: Skipped 1 previous similar message [ 2536.497174] LustreError: 77560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2541.542141] LustreError: 75808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2541.560264] LustreError: 75808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2546.456498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.586766] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.671221] LustreError: 75069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.717353] LustreError: 75069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2546.792458] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2546.924906] LustreError: 78156:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2546.932809] LustreError: 78156:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.103@tcp: cfg command failed: rc = -9 [ 2546.948939] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2546.948939] [ 2546.970937] LustreError: MGC192.168.201.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2546.992204] LustreError: 78144:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2547.000370] LustreError: 78144:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2547.011953] LustreError: 78144:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2547.023109] LustreError: 78144:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2547.058102] Lustre: server umount lustre-MDT0000 complete [ 2547.060840] LustreError: 78144:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2551.782147] LustreError: 75070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.823821] LustreError: 75070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2554.041732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.213118] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2554.525839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2554.592310] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2557.750939] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2559.988746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2560.055306] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2560.087471] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2560.150363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2570.720909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2570.727961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.737471] Lustre: Skipped 1 previous similar message [ 2570.741761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2572.265993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2572.272058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.286677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2575.338752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2575.351701] Lustre: Skipped 1 previous similar message [ 2576.819404] Lustre: server umount lustre-OST0000 complete [ 2585.568492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2585.577234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.582109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2585.611749] Lustre: Skipped 1 previous similar message [ 2585.647960] Lustre: Skipped 1 previous similar message [ 2590.690242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2590.705447] Lustre: Skipped 1 previous similar message [ 2595.297055] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2595.415409] Lustre: server umount lustre-MDT0000 complete [ 2595.811188] LustreError: 77560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.832172] LustreError: 77560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2599.708284] LustreError: 76913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783980 with bad export cookie 9729711073467558301 [ 2599.715752] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2599.723985] LustreError: 76913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2600.173509] Lustre: server umount lustre-MDT0001 complete [ 2611.433666] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2614.296711] Key type lgssc unregistered [ 2614.666326] LNet: 79942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2614.675228] LNetError: 79942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2614.689714] LNet: Removed LNI 192.168.201.103@tcp [ 2615.572792] Key type .llcrypt unregistered [ 2615.575721] Key type ._llcrypt unregistered [ 2629.905040] Key type ._llcrypt registered [ 2629.912944] Key type .llcrypt registered [ 2630.103160] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2630.399495] Key type .llcrypt unregistered [ 2630.401679] Key type ._llcrypt unregistered [ 2654.387243] Key type ._llcrypt registered [ 2654.389980] Key type .llcrypt registered [ 2654.540814] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2655.837933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2655.889987] alg: No test for adler32 (adler32-zlib) [ 2657.061333] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 2657.220054] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2658.855388] Key type lgssc registered [ 2660.143959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2667.573889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2667.594936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.951652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2673.217596] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2679.288389] LustreError: 81849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.314017] LustreError: 81849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2680.294204] LustreError: 81850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.831779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.225174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.301107] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2693.120560] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2699.433838] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2707.464041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2707.795717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.632255] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2714.997440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2722.978066] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2737.632672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2737.638361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.647089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.661677] Lustre: Skipped 1 previous similar message [ 2742.755841] LustreError: 84341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.801644] LustreError: 84341:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2743.006767] Lustre: server umount lustre-OST0000 complete [ 2747.872883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.918274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2747.920944] Lustre: Skipped 1 previous similar message [ 2752.997139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2758.112819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2758.120677] Lustre: Skipped 2 previous similar messages [ 2761.185035] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2761.323113] Lustre: server umount lustre-MDT0000 complete [ 2763.233305] LustreError: 82577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.253770] LustreError: 82577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2764.705165] LustreError: 81829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784145 with bad export cookie 13738099294367300533 [ 2764.710799] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2764.714053] LustreError: 81829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2765.038447] Lustre: server umount lustre-MDT0001 complete [ 2776.972900] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2779.645910] Key type lgssc unregistered [ 2779.941878] LNet: 85411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2779.947681] LNetError: 85411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2779.964307] LNet: Removed LNI 192.168.201.103@tcp [ 2780.997574] Key type .llcrypt unregistered [ 2781.000302] Key type ._llcrypt unregistered [ 2805.769189] Key type ._llcrypt registered [ 2805.771987] Key type .llcrypt registered [ 2805.896489] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2807.533122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2807.596891] alg: No test for adler32 (adler32-zlib) [ 2808.643674] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 2808.995980] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2810.631277] Key type lgssc registered [ 2811.837829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2821.017271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2821.038593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.533471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2827.508971] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2832.864719] LustreError: 86831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.884897] LustreError: 86831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2836.156460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.476386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2840.760211] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2847.608124] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2853.238679] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2860.128220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2860.457968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.495670] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2867.631916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2873.270258] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2881.509546] Lustre: server umount lustre-OST0000 complete [ 2887.658524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2887.666776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.695344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2893.282521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2893.293213] Lustre: Skipped 3 previous similar messages [ 2896.439124] Lustre: server umount lustre-MDT0000 complete [ 2898.401242] Lustre: 86146:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787784263/real 1787784263] req@0000000050fa4ff3 x1874627594838912/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787784279 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2898.410284] LustreError: 86826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.438164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.438172] Lustre: Skipped 1 previous similar message [ 2898.473726] LustreError: 86826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2899.488435] Lustre: 86146:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787784264/real 1787784264] req@00000000bb787c16 x1874627594839936/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787784280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2900.675391] LustreError: 86813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784281 with bad export cookie 7814154749207019478 [ 2900.682172] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2900.686431] LustreError: 86813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2901.205684] Lustre: server umount lustre-MDT0001 complete [ 2911.898291] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2914.901430] Key type lgssc unregistered [ 2915.220831] LNet: 90366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2915.230422] LNetError: 90366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2915.254875] LNet: Removed LNI 192.168.201.103@tcp [ 2916.436897] Key type .llcrypt unregistered [ 2916.443304] Key type ._llcrypt unregistered [ 2941.552383] Key type ._llcrypt registered [ 2941.555522] Key type .llcrypt registered [ 2941.690584] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2942.931262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2942.962287] alg: No test for adler32 (adler32-zlib) [ 2944.242713] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 2944.705041] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2946.431191] Key type lgssc registered [ 2947.947991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2956.415662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2956.438221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.947837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.145574] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2967.526545] LustreError: 91783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.546341] LustreError: 91783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2971.712818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.117887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2977.148471] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2984.833809] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2991.266635] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2998.839613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2999.150098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3002.235595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3006.007749] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3013.205157] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3029.998958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3030.007809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3030.033667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3033.056756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3033.061647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3033.083613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3035.671151] Lustre: server umount lustre-OST0000 complete [ 3043.299037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3043.309588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3043.313614] Lustre: Skipped 1 previous similar message [ 3043.333663] Lustre: Skipped 2 previous similar messages [ 3046.432836] Lustre: server umount lustre-MDT0000 complete [ 3048.420132] LustreError: 91777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3048.439028] LustreError: 91777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3049.901744] LustreError: 91764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784430 with bad export cookie 12737868252944724 [ 3049.908687] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3049.911732] LustreError: 91764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3050.185066] Lustre: server umount lustre-MDT0001 complete [ 3059.324382] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3062.200326] Key type lgssc unregistered [ 3062.463334] LNet: 95341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3062.472588] LNetError: 95341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3063.528053] LNet: Removed LNI 192.168.201.103@tcp [ 3064.442614] Key type .llcrypt unregistered [ 3064.444772] Key type ._llcrypt unregistered [ 3087.650125] Key type ._llcrypt registered [ 3087.652774] Key type .llcrypt registered [ 3087.749899] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3088.793080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3089.015381] alg: No test for adler32 (adler32-zlib) [ 3090.095774] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 3090.332100] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3092.023261] Key type lgssc registered [ 3093.263957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3101.843501] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3101.872546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3117.343598] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3122.463370] LustreError: 96727:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 3149.216089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3158.071967] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3167.570351] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3174.654733] Lustre: server umount lustre-OST0000 complete [ 3183.502177] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:49:24 (1787784564) [ 3190.226170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3190.790336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.426592] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3195.882503] LustreError: 98368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3195.906575] LustreError: 98368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3199.975436] LustreError: 98369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3205.092416] LustreError: 98368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.780778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.130902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3211.981307] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3219.752383] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3226.250541] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3234.047224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3234.581295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3235.713767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3242.705782] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3251.427990] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3266.544203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3266.570279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.606169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3267.558104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3267.559947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3267.618801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.263342] Lustre: server umount lustre-OST0000 complete [ 3277.793810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3277.803041] Lustre: Skipped 1 previous similar message [ 3277.810463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3277.817326] Lustre: Skipped 2 previous similar messages [ 3282.907332] Lustre: server umount lustre-MDT0000 complete [ 3282.913110] LustreError: 98368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3282.958904] LustreError: 98368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3287.016445] LustreError: 100208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.061061] LustreError: 100208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3288.329382] LustreError: 98348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784669 with bad export cookie 4986036526161880093 [ 3288.373229] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3289.003696] Lustre: server umount lustre-MDT0001 complete [ 3302.661375] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3305.593785] Key type lgssc unregistered [ 3305.905791] LNet: 102061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3305.908763] LNetError: 102061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3305.931261] LNet: Removed LNI 192.168.201.103@tcp [ 3306.797779] Key type .llcrypt unregistered [ 3306.799628] Key type ._llcrypt unregistered [ 3349.576331] Key type ._llcrypt registered [ 3349.599195] Key type .llcrypt registered [ 3349.754879] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3350.961667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3350.981200] alg: No test for adler32 (adler32-zlib) [ 3352.343916] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 3353.037831] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3354.833646] Key type lgssc registered [ 3357.058807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3368.091257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3368.122422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.708078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.932292] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3379.175638] LustreError: 104249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3379.198187] LustreError: 104249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3384.291807] LustreError: 104250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.644378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.202551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3389.441193] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3397.326191] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3403.269604] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3410.339396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3410.879511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3413.938295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3416.773611] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3425.394565] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3434.468714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3434.471983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3434.502345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3434.518564] Lustre: Skipped 1 previous similar message [ 3439.594463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3439.599033] Lustre: Skipped 1 previous similar message [ 3440.525226] Lustre: server umount lustre-OST0000 complete [ 3446.759801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3446.776849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3446.794712] Lustre: Skipped 1 previous similar message [ 3446.799952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3449.825465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.828543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3449.849078] Lustre: Skipped 1 previous similar message [ 3451.423725] Lustre: server umount lustre-MDT0000 complete [ 3454.947663] LustreError: 104244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3454.963455] LustreError: 104244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3455.401023] LustreError: 104229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784836 with bad export cookie 7703860041375249638 [ 3455.405476] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3455.413337] LustreError: 104229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3455.780159] Lustre: server umount lustre-MDT0001 complete [ 3468.155394] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3471.625554] Key type lgssc unregistered [ 3472.051632] LNet: 107786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3472.058158] LNetError: 107786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3472.081598] LNet: Removed LNI 192.168.201.103@tcp [ 3473.299791] Key type .llcrypt unregistered [ 3473.301357] Key type ._llcrypt unregistered [ 3495.990502] Key type ._llcrypt registered [ 3495.992566] Key type .llcrypt registered [ 3496.108318] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3497.381243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3497.676911] alg: No test for adler32 (adler32-zlib) [ 3498.860788] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 3499.126919] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3500.786671] Key type lgssc registered [ 3501.965783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3509.833372] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3509.855078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3525.540281] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3530.655368] LustreError: 109032:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 3557.409865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3565.227721] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3574.053614] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3581.266929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.605574] LustreError: MGC192.168.201.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3581.619587] LustreError: 110057:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3581.628227] LustreError: 110057:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3581.642666] LustreError: 110057:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3581.725837] Lustre: server umount lustre-MDT0000 complete [ 3581.728971] LustreError: 110057:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3602.399316] Lustre: 108369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787784967/real 1787784967] req@000000009d751506 x1874628318334336/t0(0) o400->MGC192.168.201.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787784983 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3602.453774] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3602.480799] LustreError: 109044:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3602.778982] Lustre: server umount lustre-OST0000 complete [ 3612.247719] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3623.076438] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3658.529237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.944106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3659.000729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3659.124304] Lustre: lustre-MDT0000: new disk, initializing [ 3659.238112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.257768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3663.834883] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3674.746271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3674.854765] Lustre: 112847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3674.902862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3674.906640] Lustre: Skipped 1 previous similar message [ 3674.973177] Lustre: lustre-MDT0001: new disk, initializing [ 3675.054307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3675.094306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3675.127815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3679.560481] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3691.006657] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3699.754818] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3709.965160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3710.451662] Lustre: lustre-OST0000: new disk, initializing [ 3710.459677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3710.480580] Lustre: 114817:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3710.612044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.172976] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3720.259081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3720.277812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3720.424799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3727.315836] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3732.183951] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3732.424701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3737.772974] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3738.002657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3741.164314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3741.171124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.173065] LustreError: Skipped 1 previous similar message [ 3741.175828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3741.205123] Lustre: Skipped 1 previous similar message [ 3741.236425] Lustre: Skipped 1 previous similar message [ 3746.279886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3746.310631] Lustre: Skipped 1 previous similar message [ 3746.699364] Lustre: server umount lustre-OST0000 complete [ 3751.396133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.431505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3756.515040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3756.526379] Lustre: Skipped 2 previous similar messages [ 3757.304269] Lustre: server umount lustre-MDT0000 complete [ 3761.746033] LustreError: 112840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785142 with bad export cookie 2850199285757216925 [ 3761.753922] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3761.757333] LustreError: 112840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3762.202778] Lustre: server umount lustre-MDT0001 complete [ 3771.109856] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:59:11 (1787785151) [ 3775.057120] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3792.594970] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3804.244265] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3846.564828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3846.873322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3846.903333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3847.007316] Lustre: lustre-MDT0000: new disk, initializing [ 3847.118233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3847.132437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3852.049428] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3863.217630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.363163] Lustre: 119723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3863.465625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3863.471259] Lustre: Skipped 1 previous similar message [ 3863.589571] Lustre: lustre-MDT0001: new disk, initializing [ 3863.773898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3863.862275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3863.875509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3868.641175] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3878.658761] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3884.479786] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3891.216113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3891.545473] Lustre: lustre-OST0000: new disk, initializing [ 3891.559943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3891.571530] Lustre: 121702:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3891.665584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.910256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3896.925758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3897.059805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3898.772947] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3909.421993] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3914.855220] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3915.118412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3920.279954] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3920.511707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3922.916185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3922.927611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3922.948819] Lustre: Skipped 1 previous similar message [ 3922.960276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3922.973858] Lustre: Skipped 1 previous similar message [ 3928.615924] Lustre: server umount lustre-OST0000 complete [ 3933.155439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.171772] Lustre: Skipped 1 previous similar message [ 3933.186099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3933.200853] Lustre: Skipped 3 previous similar messages [ 3938.927146] Lustre: server umount lustre-MDT0000 complete [ 3942.548940] LustreError: 119716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785323 with bad export cookie 2850199285757217807 [ 3942.550302] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3942.557228] LustreError: 119716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3942.873737] Lustre: server umount lustre-MDT0001 complete [ 3950.281902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3950.618079] LustreError: 123917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3950.637767] LustreError: 123917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3950.675718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3955.191173] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3955.683341] LustreError: 123918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3960.802330] LustreError: 123917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.146778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.008983] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3977.512771] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3983.276747] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3989.317048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3997.208845] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4005.013731] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4015.055988] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4021.730696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4021.739950] LustreError: Skipped 1 previous similar message [ 4021.746263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4021.758409] Lustre: Skipped 1 previous similar message [ 4021.766675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4021.773205] Lustre: Skipped 2 previous similar messages [ 4023.778158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4026.439735] Lustre: server umount lustre-OST0000 complete [ 4030.432603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4030.439594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4030.454839] Lustre: Skipped 1 previous similar message [ 4036.579128] Lustre: server umount lustre-MDT0000 complete [ 4039.143076] LustreError: 123918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.169853] LustreError: 123918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4040.411639] LustreError: 123898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785421 with bad export cookie 2850199285757218689 [ 4040.421645] LustreError: 123898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4040.423870] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4041.048616] Lustre: server umount lustre-MDT0001 complete [ 4052.562633] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4055.167248] Key type lgssc unregistered [ 4055.441926] LNet: 127551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4055.447425] LNetError: 127551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4055.461626] LNet: Removed LNI 192.168.201.103@tcp [ 4056.455653] Key type .llcrypt unregistered [ 4056.457794] Key type ._llcrypt unregistered [ 4074.036690] Key type ._llcrypt registered [ 4074.039219] Key type .llcrypt registered [ 4074.128306] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4093.072992] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4094.274830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4094.494728] alg: No test for adler32 (adler32-zlib) [ 4096.268518] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 4096.597254] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4098.319196] Key type lgssc registered [ 4099.542335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4138.262994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4138.340094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4139.598248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4139.637677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4139.740191] Lustre: lustre-MDT0000: new disk, initializing [ 4139.905508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4139.930305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4144.995469] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4155.944248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4156.110907] Lustre: 130709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4156.210341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4156.217982] Lustre: Skipped 1 previous similar message [ 4156.349261] Lustre: lustre-MDT0001: new disk, initializing [ 4156.494331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4156.537209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4156.560940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4160.877788] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4172.456695] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4181.503118] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4188.908188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4189.314261] Lustre: lustre-OST0000: new disk, initializing [ 4189.317756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4189.329686] Lustre: 132675:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4189.442140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4197.976572] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4199.010706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4199.024586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4199.183894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4207.691608] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4211.823744] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4212.089371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4216.509381] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4216.777708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4219.880064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4219.888117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4219.893975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4219.905765] Lustre: Skipped 1 previous similar message [ 4225.009114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4225.012726] Lustre: Skipped 1 previous similar message [ 4225.495907] Lustre: server umount lustre-OST0000 complete [ 4233.184530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4233.196815] LustreError: Skipped 1 previous similar message [ 4233.209540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4233.245947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4235.242093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.265585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4235.271156] Lustre: Skipped 1 previous similar message [ 4237.833846] Lustre: server umount lustre-MDT0000 complete [ 4242.202088] LustreError: 130701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785623 with bad export cookie 9137235408567119746 [ 4242.211348] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4242.215265] LustreError: 130701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4242.784449] Lustre: server umount lustre-MDT0001 complete [ 4251.873294] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 19:07:12 (1787785632) [ 4259.206375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4259.849730] LustreError: 135090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.883164] LustreError: 135090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4259.988328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.452662] LustreError: 135091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.228487] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4269.542802] LustreError: 135090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.660118] LustreError: 135091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.718376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4276.041742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4281.200811] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4289.356450] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4295.816536] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4301.281409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4301.298335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4301.305190] Lustre: Skipped 1 previous similar message [ 4304.716776] Lustre: server umount lustre-MDT0000 complete [ 4306.416305] LustreError: 135086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4306.453532] LustreError: 135086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4308.915315] LustreError: 135070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785690 with bad export cookie 9137235408567120628 [ 4308.916528] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4308.928229] LustreError: 135070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4309.519351] Lustre: server umount lustre-MDT0001 complete [ 4320.016561] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 19:08:20 (1787785700) [ 4327.242246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4342.943627] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4348.063539] LustreError: 137507:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 4373.793364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4380.542127] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4387.997817] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4390.254817] Lustre: server umount lustre-OST0000 complete [ 4397.979173] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 19:09:38 (1787785778) [ 4404.212820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.770085] LustreError: 138954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.844553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4410.004498] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4418.291797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4423.112400] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4430.945264] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4437.172894] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4443.658437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4443.916444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4443.925413] Lustre: Skipped 1 previous similar message [ 4450.217695] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4458.084917] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4470.698971] Lustre: server umount lustre-MDT0000 complete [ 4473.839597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4473.847613] LustreError: 138950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4473.865143] Lustre: Skipped 3 previous similar messages [ 4473.902862] LustreError: 138950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4474.339988] LustreError: 138934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785855 with bad export cookie 9137235408567121300 [ 4474.343460] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4474.352771] LustreError: 138934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4475.088567] Lustre: server umount lustre-MDT0001 complete [ 4489.784037] Lustre: server umount lustre-OST0000 complete [ 4500.712395] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 19:11:20 (1787785880) [ 4508.323970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.842583] LustreError: 142629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.971355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4515.101982] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4524.273894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4529.923253] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4539.565612] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4548.106380] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4557.695870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4565.017961] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4573.855264] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4578.405958] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4578.501170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4579.764924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4584.301335] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4584.520222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4589.040874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.053629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4589.072414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4589.087708] Lustre: Skipped 1 previous similar message [ 4592.902391] Lustre: server umount lustre-OST0000 complete [ 4604.384719] LustreError: 143918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.404413] LustreError: 143918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4606.111205] Lustre: server umount lustre-MDT0000 complete [ 4610.128493] LustreError: 142611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785991 with bad export cookie 9137235408567122364 [ 4610.139547] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4610.154498] LustreError: 142611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4610.598022] Lustre: server umount lustre-MDT0001 complete [ 4620.225832] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 19:13:20 (1787786000) [ 4627.539765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4643.168430] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4648.288533] LustreError: 146795:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 4674.080785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4674.088147] Lustre: Skipped 2 previous similar messages [ 4683.193972] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4693.075840] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4700.969678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4701.710902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4706.619644] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4714.207859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4720.004909] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4728.506730] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4734.396631] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4739.973884] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4740.331305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4744.715672] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4744.859849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4750.305958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4750.313037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4750.323477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4750.323486] Lustre: Skipped 2 previous similar messages [ 4750.374081] Lustre: Skipped 5 previous similar messages [ 4761.057440] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4761.368909] Lustre: server umount lustre-MDT0000 complete [ 4765.666570] LustreError: 147834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4765.689035] LustreError: 147834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4766.234824] LustreError: 146801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786147 with bad export cookie 9137235408567123393 [ 4766.253487] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4766.822282] Lustre: server umount lustre-MDT0001 complete [ 4782.073642] Lustre: server umount lustre-OST0000 complete [ 4791.380887] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 19:16:11 (1787786171) [ 4798.985633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4814.626393] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4819.743687] LustreError: 150993:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 4851.969854] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4862.814725] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4870.479697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4871.080505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4875.560441] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4884.105207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4888.651798] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4895.417174] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4900.888772] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4907.870788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4908.100586] Lustre: lustre-OST0001: new disk, initializing [ 4908.103735] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4908.108353] Lustre: 153867:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4913.956557] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4916.304053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4916.311472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4916.390839] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4922.233313] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4927.025972] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4927.292574] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4931.837493] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4932.021601] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4935.649096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4935.651920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4935.680190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4935.682902] Lustre: Skipped 8 previous similar messages [ 4939.826534] Lustre: server umount lustre-OST0000 complete [ 4947.431414] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4949.632694] Lustre: server umount lustre-OST0001 complete [ 4960.734736] Lustre: server umount lustre-MDT0000 complete [ 4965.433700] LustreError: 151000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786346 with bad export cookie 9137235408567124345 [ 4965.436304] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4965.464819] LustreError: 151000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4966.021994] Lustre: server umount lustre-MDT0001 complete [ 4992.458512] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 19:19:32 (1787786372) [ 4994.190870] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4996.454929] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4998.409058] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 19:19:38 (1787786378) [ 5005.066332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.272613] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5005.297164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5005.496354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5005.504704] Lustre: Skipped 6 previous similar messages [ 5009.951763] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5017.993940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5018.113387] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5018.153211] Lustre: 157427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5023.615884] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5031.740410] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5036.677803] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5043.694189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5043.996578] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5051.107955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5051.663205] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5060.725675] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5065.818296] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5066.062986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5071.174304] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5071.409747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5076.448493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5076.454302] LustreError: Skipped 2 previous similar messages [ 5076.459459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5076.470377] Lustre: Skipped 5 previous similar messages [ 5076.474577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5076.479328] Lustre: Skipped 7 previous similar messages [ 5079.700092] Lustre: server umount lustre-OST0000 complete [ 5101.269616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5107.539681] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5115.606803] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5116.970684] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.976936] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5116.978707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.935274] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5121.142259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5126.555463] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5126.898499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5133.260322] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9ff4c8b58800.ost_server_uuid 50 [ 5134.579509] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ff4c8b58800.ost_server_uuid in FULL state after 0 sec [ 5134.876342] LustreError: 158336:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.3@tcp) returned error from blocking AST (req@0000000052ca6d57 x1874628944600704 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000fa522bdd/0x7ecdfb3164c3cad3 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.3@tcp remote: 0x92b8da73a2c3bf27 expref: 5 pid: 157432 timeout: 5234 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5134.945313] LustreError: lustre-MDT0000: A client on nid 192.168.201.3@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5134.966057] LustreError: 157422:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.3@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000fa522bdd/0x7ecdfb3164c3cad3 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.3@tcp remote: 0x92b8da73a2c3bf27 expref: 6 pid: 157432 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5140.967569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5140.987120] LustreError: Skipped 1 previous similar message [ 5146.256939] Lustre: server umount lustre-OST0000 complete [ 5157.857833] LustreError: 157436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.867888] LustreError: 157436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5157.903349] Lustre: server umount lustre-MDT0000 complete [ 5161.180418] LustreError: 157418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786542 with bad export cookie 9137235408567125528 [ 5161.205545] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.206814] LustreError: 157418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5167.707756] Lustre: server umount lustre-MDT0001 complete [ 5179.212683] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5182.879161] Key type lgssc unregistered [ 5183.256939] LNet: 163546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5183.269426] LNetError: 163546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5184.299543] LNet: Removed LNI 192.168.201.103@tcp [ 5185.529460] Key type .llcrypt unregistered [ 5185.532698] Key type ._llcrypt unregistered [ 5214.073821] Key type ._llcrypt registered [ 5214.075868] Key type .llcrypt registered [ 5214.280457] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5216.418441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5216.570527] alg: No test for adler32 (adler32-zlib) [ 5217.783930] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 5218.017079] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5219.687223] Key type lgssc registered [ 5221.359119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5230.553615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5230.578404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5232.324977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5238.225899] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5242.854832] LustreError: 164964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5242.867942] LustreError: 164964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5246.781134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.170319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5251.832877] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5259.117536] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5265.604957] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5272.553684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5272.924580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5277.046173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5279.128600] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5286.732854] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5297.104640] Lustre: Failing over lustre-MDT0000 [ 5297.402082] Lustre: server umount lustre-MDT0000 complete [ 5297.648109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.658707] LustreError: 164960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5297.665861] Lustre: Skipped 2 previous similar messages [ 5297.718314] LustreError: 164960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5302.752929] LustreError: 164960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5302.770954] LustreError: 164960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5307.874437] LustreError: 164959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.906256] LustreError: 164959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5309.838297] LustreError: 164945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.3@tcp arrived at 1787786690 with bad export cookie 16822967160231635508 [ 5309.866306] LustreError: 164945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5313.011996] LustreError: 164965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5313.044033] LustreError: 164965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5317.881987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5318.050587] Lustre: 164275:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786683/real 1787786683] req@00000000fde94f8f x1874630120859392/t0(0) o400->MGC192.168.201.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787786699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5318.051237] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5318.395287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5318.449041] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5322.976251] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5323.752434] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5323.765244] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5325.344366] Lustre: lustre-MDT0000: Denying connection for new client d5399240-27c5-4c26-9c75-567e681896e7 (at 192.168.201.3@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5333.472847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5333.476847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5333.485529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5334.007450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5339.105871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5339.596876] Lustre: server umount lustre-OST0000 complete [ 5343.858218] LustreError: 168692:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5343.869854] Lustre: 168042:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5343.883929] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5343.918441] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5343.931239] LustreError: 168042:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000833fa3ca x1874630120875776/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5343.952845] LustreError: 168042:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5343.966915] LustreError: 168042:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5343.979757] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5344.234156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.267835] Lustre: Skipped 1 previous similar message [ 5344.276153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5344.280718] Lustre: Skipped 1 previous similar message [ 5344.370848] Lustre: server umount lustre-MDT0000 complete [ 5348.742756] LustreError: 164943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786729 with bad export cookie 16822967160231635613 [ 5348.751591] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5348.762720] LustreError: 164943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5348.784442] LustreError: 164273:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5351.229051] Lustre: server umount lustre-MDT0001 complete [ 5361.354918] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5363.994516] Key type lgssc unregistered [ 5364.227683] LNet: 169448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5364.233612] LNetError: 169448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5364.264166] LNet: Removed LNI 192.168.201.103@tcp [ 5365.388306] Key type .llcrypt unregistered [ 5365.397693] Key type ._llcrypt unregistered [ 5392.287795] Key type ._llcrypt registered [ 5392.290652] Key type .llcrypt registered [ 5392.447587] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5393.618147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5393.677297] alg: No test for adler32 (adler32-zlib) [ 5394.771481] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 5394.980059] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5396.655172] Key type lgssc registered [ 5397.716489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5405.036630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5405.051460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.529258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5410.474740] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5416.929366] LustreError: 170861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5416.943642] LustreError: 170861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5418.916921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5419.480717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5424.016252] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5431.365928] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5436.638623] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5444.557657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5444.852312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5451.393844] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5452.026938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5460.328677] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5469.673824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.680628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5469.700844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5474.459506] Lustre: server umount lustre-OST0000 complete [ 5479.906088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.923712] Lustre: Skipped 1 previous similar message [ 5479.929807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5479.940279] Lustre: Skipped 2 previous similar messages [ 5485.040196] LustreError: 171872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5485.064542] LustreError: 171872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5485.335182] Lustre: server umount lustre-MDT0000 complete [ 5489.121285] LustreError: 170862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5489.140051] LustreError: 170862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5489.770662] LustreError: 170843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786870 with bad export cookie 4136683686436151993 [ 5489.785659] LustreError: 170843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5489.790539] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5490.203911] Lustre: server umount lustre-MDT0001 complete [ 5505.745324] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5509.595752] Key type lgssc unregistered [ 5509.834819] LNet: 174396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5509.839142] LNetError: 174396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5509.857475] LNet: Removed LNI 192.168.201.103@tcp [ 5511.037957] Key type .llcrypt unregistered [ 5511.039640] Key type ._llcrypt unregistered [ 5548.192367] Key type ._llcrypt registered [ 5548.194352] Key type .llcrypt registered [ 5548.271674] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5549.608729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5549.627165] alg: No test for adler32 (adler32-zlib) [ 5550.749729] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 5550.986131] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5552.671179] Key type lgssc registered [ 5553.955584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5561.803852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5561.817370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.266850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5567.581896] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5573.602079] LustreError: 176100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.628525] LustreError: 176100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5575.722494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5576.133050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5580.728971] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5588.919991] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5594.978813] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5602.638488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5603.113961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5608.260129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5611.165755] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5620.289856] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5628.896373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5628.909472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.924321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5631.969597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5631.983152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5632.005756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5632.036757] Lustre: Skipped 1 previous similar message [ 5633.821446] Lustre: server umount lustre-OST0000 complete [ 5642.213171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.225488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5642.256692] Lustre: Skipped 1 previous similar message [ 5642.273975] Lustre: Skipped 1 previous similar message [ 5644.747563] Lustre: server umount lustre-MDT0000 complete [ 5647.347220] LustreError: 178542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5647.396753] LustreError: 178542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5649.022339] LustreError: 176082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787030 with bad export cookie 14993312467886494069 [ 5649.031489] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5649.356935] Lustre: server umount lustre-MDT0001 complete [ 5660.199323] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5663.362536] Key type lgssc unregistered [ 5663.670981] LNet: 179635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5663.677589] LNetError: 179635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5664.744632] LNet: Removed LNI 192.168.201.103@tcp [ 5665.774860] Key type .llcrypt unregistered [ 5665.776684] Key type ._llcrypt unregistered [ 5692.800991] Key type ._llcrypt registered [ 5692.803760] Key type .llcrypt registered [ 5692.918613] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5694.180356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5694.288577] alg: No test for adler32 (adler32-zlib) [ 5695.376444] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 5695.642370] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5697.383216] Key type lgssc registered [ 5698.769885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5712.094159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5712.124915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5713.554422] Lustre: *** cfs_fail_loc=135, val=0*** [ 5713.634170] LustreError: 181162:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5713.639231] LustreError: 181162:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.103@tcp: cfg command failed: rc = -2 [ 5713.655630] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5713.655630] [ 5713.663817] LustreError: MGC192.168.201.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5713.677687] LustreError: 181123:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5713.684627] LustreError: 181123:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5713.702629] LustreError: 181123:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5713.716578] LustreError: 181123:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5713.861379] Lustre: server umount lustre-MDT0000 complete [ 5713.864511] LustreError: 181123:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5721.533737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.059471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5726.583949] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5732.321350] LustreError: 181534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5732.334826] LustreError: 181534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5735.317913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5735.733768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5740.812531] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5748.580644] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5755.005376] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5760.609209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5760.630108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5760.637017] Lustre: Skipped 1 previous similar message [ 5760.656454] Lustre: Skipped 1 previous similar message [ 5765.928814] Lustre: server umount lustre-MDT0000 complete [ 5766.121985] LustreError: 182527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.061938] LustreError: 181516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787151 with bad export cookie 18374737023870878213 [ 5770.071500] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5770.083126] LustreError: 181516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5770.414622] Lustre: server umount lustre-MDT0001 complete [ 5782.796389] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5786.068079] Key type lgssc unregistered [ 5786.483633] LNet: 183858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5786.505428] LNetError: 183858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5786.547813] LNet: Removed LNI 192.168.201.103@tcp [ 5787.559682] Key type .llcrypt unregistered [ 5787.561846] Key type ._llcrypt unregistered [ 5810.011192] Key type ._llcrypt registered [ 5810.013668] Key type .llcrypt registered [ 5810.122750] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5810.479346] Key type .llcrypt unregistered [ 5810.481553] Key type ._llcrypt unregistered [ 5827.783421] Key type ._llcrypt registered [ 5827.786489] Key type .llcrypt registered [ 5827.923743] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5829.525265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5829.621506] alg: No test for adler32 (adler32-zlib) [ 5830.788657] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 5831.167379] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5832.881088] Key type lgssc registered [ 5834.255706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5841.956945] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5841.988567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5857.631398] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5862.751487] LustreError: 186010:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 5889.503718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.320881] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5906.150733] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5912.273581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.032924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5913.088846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5917.992444] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5918.178261] LustreError: 186035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5918.204750] LustreError: 186035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5923.303657] LustreError: 186036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.324124] LustreError: 187047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5927.885590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.513225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5933.723825] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5941.338932] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5947.529969] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5950.653438] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5964.260562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5964.267055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5964.283153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5967.787996] Lustre: server umount lustre-OST0000 complete [ 5974.498819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5974.498970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5974.513927] LustreError: Skipped 1 previous similar message [ 5974.515714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5974.541703] Lustre: Skipped 2 previous similar messages [ 5974.572193] Lustre: Skipped 2 previous similar messages [ 5977.847158] Lustre: server umount lustre-MDT0000 complete [ 5979.625029] LustreError: 187046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5979.659559] LustreError: 187046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5981.332812] LustreError: 186018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787362 with bad export cookie 12761219312200568295 [ 5981.347541] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5981.977600] Lustre: server umount lustre-MDT0001 complete [ 5993.650730] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5996.847394] Key type lgssc unregistered [ 5997.201613] LNet: 189975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5997.207444] LNetError: 189975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5997.219296] LNet: Removed LNI 192.168.201.103@tcp [ 5998.235606] Key type .llcrypt unregistered [ 5998.237614] Key type ._llcrypt unregistered [ 6024.450719] Key type ._llcrypt registered [ 6024.452868] Key type .llcrypt registered [ 6024.662787] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6026.683518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6026.716750] alg: No test for adler32 (adler32-zlib) [ 6027.899436] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 6028.119745] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6029.903826] Key type lgssc registered [ 6031.566791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6039.946257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6039.968912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6041.724556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6046.884385] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6051.300109] LustreError: 191390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6051.332831] LustreError: 191390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6056.407645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.425076] LustreError: 191391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6056.955612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6061.879694] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6071.506151] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6077.858654] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6086.805943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6087.068071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6095.156603] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6104.144375] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6111.679054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6120.652041] Lustre: Failing over lustre-MDT0000 [ 6121.013040] Lustre: server umount lustre-MDT0000 complete [ 6121.312813] LustreError: 191387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6121.961383] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.985488] Lustre: Skipped 1 previous similar message [ 6126.393898] LustreError: 191387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6126.422802] LustreError: 191387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6131.508701] LustreError: 191385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6131.534948] LustreError: 191385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6138.336264] Lustre: 190702:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787787503/real 1787787503] req@000000008f1c3d58 x1874630970209408/t0(0) o400->MGC192.168.201.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787787519 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6138.348805] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6140.961035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6141.620821] LustreError: 191385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6141.646290] LustreError: 191385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6148.576137] LustreError: 190699:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bd9d1242 x1874630970214528/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6149.041185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6149.184719] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6153.019042] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6154.236781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6154.251577] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6154.294087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6154.546218] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6159.888414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6170.082491] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6179.824045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6179.845676] Lustre: Skipped 2 previous similar messages [ 6179.849196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6184.934871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6184.950339] Lustre: Skipped 2 previous similar messages [ 6185.858925] Lustre: server umount lustre-OST0000 complete [ 6195.169799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.179145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6195.196301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6196.760594] Lustre: server umount lustre-MDT0000 complete [ 6200.290477] LustreError: 191390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6200.320094] LustreError: 191390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6200.946216] LustreError: 191370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787582 with bad export cookie 1776732681627113018 [ 6200.975260] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6201.378448] Lustre: server umount lustre-MDT0001 complete [ 6215.089724] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6218.553731] Key type lgssc unregistered [ 6218.978831] LNet: 196359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6218.985483] LNetError: 196359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6218.996969] LNet: Removed LNI 192.168.201.103@tcp [ 6220.048602] Key type .llcrypt unregistered [ 6220.050997] Key type ._llcrypt unregistered [ 6247.176345] Key type ._llcrypt registered [ 6247.178735] Key type .llcrypt registered [ 6247.276759] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6248.604852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6248.638099] alg: No test for adler32 (adler32-zlib) [ 6249.895417] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 6250.242521] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6252.056352] Key type lgssc registered [ 6253.636758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6261.426274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6261.451152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.009211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6268.444934] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6273.506723] LustreError: 197772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6273.536393] LustreError: 197772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6277.640432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6278.138406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6283.838623] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6293.225590] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6299.371612] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6307.275669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6307.865589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6308.933745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6315.296510] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6326.044359] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6330.159303] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6335.440467] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6353.496612] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6359.012041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6359.021146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6359.022295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6364.131401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6364.139270] Lustre: Skipped 1 previous similar message [ 6364.650241] Lustre: server umount lustre-OST0000 complete [ 6369.253612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6369.265339] Lustre: Skipped 1 previous similar message [ 6369.273445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6369.279340] Lustre: Skipped 2 previous similar messages [ 6374.375794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6374.387860] Lustre: Skipped 1 previous similar message [ 6379.490996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6379.509985] Lustre: Skipped 1 previous similar message [ 6383.071720] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6383.558881] Lustre: server umount lustre-MDT0000 complete [ 6384.612470] LustreError: 197769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6384.635843] LustreError: 197769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6388.733769] LustreError: 198510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787769 with bad export cookie 997124859090365502 [ 6388.737603] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6388.748694] LustreError: 198510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6389.272793] Lustre: server umount lustre-MDT0001 complete [ 6402.224072] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6405.635331] Key type lgssc unregistered [ 6405.936711] LNet: 201645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6405.949644] LNetError: 201645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6405.964801] LNet: Removed LNI 192.168.201.103@tcp [ 6406.904911] Key type .llcrypt unregistered [ 6406.907176] Key type ._llcrypt unregistered [ 6426.276911] Key type ._llcrypt registered [ 6426.280729] Key type .llcrypt registered [ 6426.531543] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6428.425485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6428.441354] alg: No test for adler32 (adler32-zlib) [ 6429.570824] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 6429.965417] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6431.655478] Key type lgssc registered [ 6433.254796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6443.242638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6443.289902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6444.891124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.793434] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6454.242872] LustreError: 202912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6454.272720] LustreError: 202912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6459.367421] LustreError: 202913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6460.554975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6461.239315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6466.932192] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6475.412152] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6482.646281] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6488.896669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6489.139390] LustreError: 204736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6489.353173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6494.720558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6497.642295] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6507.922356] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6514.094949] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6526.740451] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6540.006804] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6552.944189] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6566.368150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6566.377722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6566.411173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6567.392708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6567.401281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6567.405855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6570.673602] Lustre: server umount lustre-OST0000 complete [ 6577.638927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6577.643463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6577.663527] Lustre: Skipped 1 previous similar message [ 6577.668654] Lustre: Skipped 1 previous similar message [ 6582.753191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6582.776938] Lustre: Skipped 2 previous similar messages [ 6587.880425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6587.893479] Lustre: Skipped 1 previous similar message [ 6589.410971] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6589.649372] Lustre: server umount lustre-MDT0000 complete [ 6592.994917] LustreError: 202912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.210144] LustreError: 206830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787975 with bad export cookie 12723704146908103906 [ 6594.217670] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6594.219907] LustreError: 206830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6594.909407] Lustre: server umount lustre-MDT0001 complete [ 6606.185954] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6611.160107] Key type lgssc unregistered [ 6611.638617] LNet: 208037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6611.658164] LNetError: 208037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6611.673086] LNet: Removed LNI 192.168.201.103@tcp [ 6613.146131] Key type .llcrypt unregistered [ 6613.148320] Key type ._llcrypt unregistered [ 6641.891229] Key type ._llcrypt registered [ 6641.896338] Key type .llcrypt registered [ 6642.060445] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6643.600597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6643.626894] alg: No test for adler32 (adler32-zlib) [ 6644.794946] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 6645.136664] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6646.879790] Key type lgssc registered [ 6648.135822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6656.572462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6656.590230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.115157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6663.297913] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6668.265958] LustreError: 209450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6668.302854] LustreError: 209450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6673.188882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.379987] LustreError: 209451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6673.709650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6678.712279] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6685.602388] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6690.690780] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6697.164295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6697.526826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6703.797108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6704.677498] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6714.158789] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6729.706231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6729.706916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.709589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6729.778369] Lustre: Skipped 1 previous similar message [ 6734.824980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6734.833158] Lustre: Skipped 1 previous similar message [ 6735.726426] Lustre: server umount lustre-OST0000 complete [ 6740.450595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6740.463615] LustreError: Skipped 1 previous similar message [ 6740.474681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6740.493070] Lustre: Skipped 1 previous similar message [ 6740.501225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6745.058408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6745.071471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.570947] Lustre: server umount lustre-MDT0000 complete [ 6750.181101] LustreError: 209446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.195593] LustreError: 209446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6750.917200] LustreError: 209430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788132 with bad export cookie 13634139892915363485 [ 6750.930631] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6750.939203] LustreError: 209430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6751.615383] Lustre: server umount lustre-MDT0001 complete [ 6764.737871] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6768.754904] Key type lgssc unregistered [ 6769.093711] LNet: 213136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6769.109428] LNetError: 213136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6769.125172] LNet: Removed LNI 192.168.201.103@tcp [ 6770.133832] Key type .llcrypt unregistered [ 6770.135766] Key type ._llcrypt unregistered [ 6790.326287] Key type ._llcrypt registered [ 6790.328669] Key type .llcrypt registered [ 6790.440768] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6791.705041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6791.789731] alg: No test for adler32 (adler32-zlib) [ 6793.039845] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 6793.380836] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6795.159815] Key type lgssc registered [ 6796.392584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6805.138820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6805.162979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6806.714136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6812.045651] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6816.234846] LustreError: 214407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6816.257753] LustreError: 214407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6821.344502] LustreError: 214406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6821.382610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6821.846444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6826.336764] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6834.397136] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6842.206417] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6848.739596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6849.312396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6851.391048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6856.584471] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6864.518501] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6882.272817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6882.282956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6882.294880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6884.182811] Lustre: server umount lustre-OST0000 complete [ 6888.416166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6888.419542] LustreError: Skipped 1 previous similar message [ 6888.429986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6888.443915] Lustre: Skipped 1 previous similar message [ 6888.455058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6888.460474] Lustre: Skipped 1 previous similar message [ 6892.517266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6892.517857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6894.139622] Lustre: server umount lustre-MDT0000 complete [ 6897.632490] LustreError: 215880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6897.655971] LustreError: 215880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6898.576844] LustreError: 214386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788279 with bad export cookie 3870068873672442871 [ 6898.603404] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6898.605837] LustreError: 214386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6899.172574] Lustre: server umount lustre-MDT0001 complete [ 6911.888459] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6916.034690] Key type lgssc unregistered [ 6916.590146] LNet: 218092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6916.593854] LNetError: 218092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6916.612802] LNet: Removed LNI 192.168.201.103@tcp [ 6918.060669] Key type .llcrypt unregistered [ 6918.062584] Key type ._llcrypt unregistered [ 6939.771915] Key type ._llcrypt registered [ 6939.774892] Key type .llcrypt registered [ 6939.855212] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6941.045589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6941.072286] alg: No test for adler32 (adler32-zlib) [ 6942.188876] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 6942.446976] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6944.119188] Key type lgssc registered [ 6945.434550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6954.552872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6954.600931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6956.324090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6961.024952] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6966.754315] LustreError: 219358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6966.774148] LustreError: 219358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6969.583910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6969.936282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6974.560675] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6981.269893] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6986.375492] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6993.246343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6993.520827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7000.511710] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7001.728506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7008.309449] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7017.813973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7017.888802] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7018.184951] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7024.310092] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7032.929064] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7034.835737] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7046.260563] Lustre: Permanently deactivating lustre-OST0001 [ 7046.282705] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7053.407391] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7064.366475] Lustre: Permanently reactivating lustre-OST0001 [ 7064.390816] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7064.400778] Lustre: Skipped 2 previous similar messages [ 7072.375686] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7072.399715] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7072.424605] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7072.446187] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7072.492822] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7077.856099] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7077.858388] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.869798] LustreError: Skipped 1 previous similar message [ 7077.870960] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7077.900840] Lustre: Skipped 2 previous similar messages [ 7080.426737] Lustre: server umount lustre-OST0001 complete [ 7082.984425] LustreError: 221180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7083.009353] LustreError: 221180:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7087.075106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7087.092826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.112678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7087.123557] Lustre: Skipped 1 previous similar message [ 7088.104217] LustreError: 221250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.128140] LustreError: 221250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7090.389785] Lustre: server umount lustre-OST0000 complete [ 7097.824147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7097.839937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.869739] Lustre: Skipped 1 previous similar message [ 7097.880353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7097.886137] Lustre: Skipped 2 previous similar messages [ 7101.380456] Lustre: server umount lustre-MDT0000 complete [ 7103.460539] LustreError: 219358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7103.484676] LustreError: 219358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7105.876119] LustreError: 219339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788486 with bad export cookie 17514318985750119323 [ 7105.896993] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7105.902537] LustreError: 219339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7106.473855] Lustre: server umount lustre-MDT0001 complete [ 7119.705510] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7122.501354] Key type lgssc unregistered [ 7122.905642] LNet: 224427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7122.916251] LNetError: 224427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7122.939579] LNet: Removed LNI 192.168.201.103@tcp [ 7124.069103] Key type .llcrypt unregistered [ 7124.072666] Key type ._llcrypt unregistered [ 7161.914955] Key type ._llcrypt registered [ 7161.918788] Key type .llcrypt registered [ 7162.068759] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7163.884382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7163.974089] alg: No test for adler32 (adler32-zlib) [ 7165.250507] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 7165.703739] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7167.487252] Key type lgssc registered [ 7168.985724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7177.603849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7177.617367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7178.809478] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7178.825673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7178.959224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7183.255280] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7193.654366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7193.721253] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7193.746552] Lustre: 226631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7193.950766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7198.597133] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7206.500163] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7212.651597] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7220.616299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7220.775323] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7220.996879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7226.969970] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7229.012289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7235.146598] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7239.196608] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7245.971511] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7256.172235] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7260.531829] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7270.463714] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7293.906575] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7293.916103] Lustre: Skipped 2 previous similar messages [ 7331.207282] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7331.212219] Lustre: Skipped 4 previous similar messages [ 7399.904130] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7399.916239] Lustre: Skipped 7 previous similar messages [ 7418.337229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7418.344132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7418.355021] LustreError: Skipped 1 previous similar message [ 7418.355612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7418.392913] Lustre: Skipped 1 previous similar message [ 7422.591206] Lustre: server umount lustre-OST0000 complete [ 7428.576045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7428.590164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7428.604027] Lustre: Skipped 1 previous similar message [ 7428.607893] Lustre: Skipped 1 previous similar message [ 7432.937592] Lustre: server umount lustre-MDT0000 complete [ 7436.062691] LustreError: 226625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788817 with bad export cookie 16571207792760835952 [ 7436.074644] LustreError: 226625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7436.076494] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7436.368917] Lustre: server umount lustre-MDT0001 complete [ 7446.819536] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7450.272840] Key type lgssc unregistered [ 7450.567558] LNet: 231417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7450.591484] LNetError: 231417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7450.605359] LNet: Removed LNI 192.168.201.103@tcp [ 7451.381283] Key type .llcrypt unregistered [ 7451.383061] Key type ._llcrypt unregistered [ 7480.453523] Key type ._llcrypt registered [ 7480.456683] Key type .llcrypt registered [ 7480.537746] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7481.693818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7481.811080] alg: No test for adler32 (adler32-zlib) [ 7482.862780] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 7483.133791] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7484.799187] Key type lgssc registered [ 7485.985469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7492.981704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7493.000485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7494.446844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7498.217880] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7504.882250] LustreError: 232842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7504.900186] LustreError: 232842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7506.317748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7506.880848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7511.381312] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7518.615588] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7524.303497] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7530.875808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7531.126521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7537.393022] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7539.401235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7545.463720] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7563.233595] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7563.257289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7563.299871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7564.179287] Lustre: server umount lustre-OST0000 complete [ 7568.365189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7568.379797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7568.408460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7570.405387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7570.409702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7575.521412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7575.527815] Lustre: Skipped 2 previous similar messages [ 7578.643668] Lustre: server umount lustre-MDT0000 complete [ 7580.641215] LustreError: 232843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7580.652712] LustreError: 232843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7582.598586] LustreError: 235422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788963 with bad export cookie 17215489459298348829 [ 7582.609043] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7582.620571] LustreError: 235422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7582.954139] Lustre: server umount lustre-MDT0001 complete [ 7595.283882] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7597.874931] Key type lgssc unregistered [ 7598.259983] LNet: 236534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7598.267937] LNetError: 236534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7598.283172] LNet: Removed LNI 192.168.201.103@tcp [ 7598.943711] Key type .llcrypt unregistered [ 7598.947218] Key type ._llcrypt unregistered [ 7623.184589] Key type ._llcrypt registered [ 7623.186968] Key type .llcrypt registered [ 7623.268942] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7624.168632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7624.234450] alg: No test for adler32 (adler32-zlib) [ 7625.249318] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 7625.479421] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7627.151188] Key type lgssc registered [ 7628.434300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7899.916563] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7902.868717] Key type lgssc unregistered [ 7903.204088] LNet: 238563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7903.209580] LNetError: 238563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7903.233293] LNet: Removed LNI 192.168.201.103@tcp [ 7904.093209] Key type .llcrypt unregistered [ 7904.096820] Key type ._llcrypt unregistered [ 7930.644391] Key type ._llcrypt registered [ 7930.648469] Key type .llcrypt registered [ 7930.755473] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 7947.177941] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7948.822539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7948.893989] alg: No test for adler32 (adler32-zlib) [ 7949.886922] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 7950.107546] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7951.743344] Key type lgssc registered [ 7953.060051] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7983.514401] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7686 sec ========== 20:09:23 (1787789363) [ 7985.286614] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:09:25 (1787789365) === [ 7988.377337] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:09:29 (1787789369) === [ 8024.999937] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8027.600385] Key type lgssc unregistered [ 8027.832590] LNet: 243806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8027.838611] LNetError: 243806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8027.850503] LNet: Removed LNI 192.168.201.103@tcp [ 8028.493992] Key type .llcrypt unregistered [ 8028.497350] Key type ._llcrypt unregistered