[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 640562067 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2544MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002364] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007536] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010147] LSM: Security Framework initializing [ 0.011063] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.022553] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027494] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031120] *** VALIDATE tmpfs *** [ 0.032466] *** VALIDATE proc *** [ 0.034111] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038116] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044441] debug: unmapping init [mem 0xffffffff92059000-0xffffffff92060fff] [ 0.046194] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047783] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053017] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.055287] rcu: Hierarchical SRCU implementation. [ 0.057383] smp: Bringing up secondary CPUs ... [ 0.058604] x86: Booting SMP configuration: [ 0.059026] .... node #0, CPUs: #1 #2 #3 [ 0.062546] smp: Brought up 1 node, 4 CPUs [ 0.064022] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151559] node 0 deferred pages initialised in 84ms [ 0.155130] devtmpfs: initialized [ 0.156237] x86/mm: Memory block size: 128MB [ 0.158800] gcov: version magic: 0x41383552 [ 0.160327] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162315] pinctrl core: initialized pinctrl subsystem [ 0.163181] [ 0.163716] ************************************************************* [ 0.164011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165016] ** ** [ 0.166011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167011] ** ** [ 0.168011] ** This means that this kernel is built to expose internal ** [ 0.169011] ** IOMMU data structures, which may compromise security on ** [ 0.170011] ** your system. ** [ 0.171013] ** ** [ 0.172011] ** If you see this message and you are not debugging the ** [ 0.173012] ** kernel, report this immediately to your vendor! ** [ 0.174014] ** ** [ 0.175014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176012] ************************************************************* [ 0.177820] NET: Registered protocol family 16 [ 0.178434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181480] cpuidle: using governor menu [ 0.183835] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186531] PCI: Using configuration type 1 for base access [ 0.187132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.198034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201031] cryptd: max_cpu_qlen set to 1000 [ 0.206698] ACPI: Added _OSI(Module Device) [ 0.208026] ACPI: Added _OSI(Processor Device) [ 0.210015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.211013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.216028] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.222588] ACPI: Interpreter enabled [ 0.224055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.225009] ACPI: Using IOAPIC for interrupt routing [ 0.226090] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.229399] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239353] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.242041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.247088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.253282] acpiphp: Slot [2] registered [ 0.255089] acpiphp: Slot [5] registered [ 0.256080] acpiphp: Slot [6] registered [ 0.257153] acpiphp: Slot [7] registered [ 0.258121] acpiphp: Slot [8] registered [ 0.260119] acpiphp: Slot [9] registered [ 0.261123] acpiphp: Slot [10] registered [ 0.262106] acpiphp: Slot [3] registered [ 0.264131] acpiphp: Slot [4] registered [ 0.265099] acpiphp: Slot [11] registered [ 0.267128] acpiphp: Slot [12] registered [ 0.269099] acpiphp: Slot [13] registered [ 0.270083] acpiphp: Slot [14] registered [ 0.271097] acpiphp: Slot [15] registered [ 0.273104] acpiphp: Slot [16] registered [ 0.274096] acpiphp: Slot [17] registered [ 0.276119] acpiphp: Slot [18] registered [ 0.277098] acpiphp: Slot [19] registered [ 0.278105] acpiphp: Slot [20] registered [ 0.280089] acpiphp: Slot [21] registered [ 0.281115] acpiphp: Slot [22] registered [ 0.283098] acpiphp: Slot [23] registered [ 0.284145] acpiphp: Slot [24] registered [ 0.286099] acpiphp: Slot [25] registered [ 0.287105] acpiphp: Slot [26] registered [ 0.289110] acpiphp: Slot [27] registered [ 0.290089] acpiphp: Slot [28] registered [ 0.291116] acpiphp: Slot [29] registered [ 0.293106] acpiphp: Slot [30] registered [ 0.294099] acpiphp: Slot [31] registered [ 0.296062] PCI host bridge to bus 0000:00 [ 0.298030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.300027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.303027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.306024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.308034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.311055] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.313270] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.317102] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.320352] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.330775] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.337069] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.340019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.348658] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.349855] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.352041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.355949] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.361013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.375016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.380729] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.387228] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.480020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.521020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.541019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.554469] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.565016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.587024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.672020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.707768] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.722018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.735023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.759016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.773153] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.790016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.811017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.836018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.847116] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.862017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.872019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.894016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.903000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.909014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.913017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.929013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.936651] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.939360] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.941335] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.944376] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.946227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.951120] iommu: Default domain type: Passthrough [ 0.953396] SCSI subsystem initialized [ 0.954070] ACPI: bus type USB registered [ 0.954877] usbcore: registered new interface driver usbfs [ 0.956074] usbcore: registered new interface driver hub [ 0.958061] usbcore: registered new device driver usb [ 0.959116] pps_core: LinuxPPS API ver. 1 registered [ 0.960007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.963050] PTP clock support registered [ 0.964153] EDAC MC: Ver: 3.0.0 [ 0.966142] PCI: Using ACPI for IRQ routing [ 0.968903] NetLabel: Initializing [ 0.970010] NetLabel: domain hash size = 128 [ 0.971008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.973086] NetLabel: unlabeled traffic allowed by default [ 0.975150] vgaarb: loaded [ 0.977273] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.979014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.987334] clocksource: Switched to clocksource kvm-clock [ 1.103198] VFS: Disk quotas dquot_6.6.0 [ 1.104874] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.107531] *** VALIDATE ramfs *** [ 1.108834] *** VALIDATE hugetlbfs *** [ 1.110621] pnp: PnP ACPI init [ 1.113232] pnp: PnP ACPI: found 6 devices [ 1.130110] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.134068] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.136514] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.138608] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.140903] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.143264] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.146195] NET: Registered protocol family 2 [ 1.148724] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.153319] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.157329] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.163445] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.167076] TCP: Hash tables configured (established 65536 bind 65536) [ 1.170236] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.173567] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.176545] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.179653] NET: Registered protocol family 1 [ 1.182332] RPC: Registered named UNIX socket transport module. [ 1.184836] RPC: Registered udp transport module. [ 1.186696] RPC: Registered tcp transport module. [ 1.188685] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.191304] NET: Registered protocol family 44 [ 1.193037] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.195524] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.199072] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.201956] PCI: CLS 0 bytes, default 64 [ 1.204338] Unpacking initramfs... [ 2.597864] debug: unmapping init [mem 0xffff963e3cc54000-0xffff963e3ffbffff] [ 2.601804] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.604121] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.607100] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.116933] Initialise system trusted keyrings [ 3.118072] Key type blacklist registered [ 3.119715] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.125878] zbud: loaded [ 3.129553] *** VALIDATE nfs *** [ 3.130399] *** VALIDATE nfs4 *** [ 3.131666] pstore: using deflate compression [ 3.134517] Platform Keyring initialized [ 3.241964] NET: Registered protocol family 38 [ 3.243267] Key type asymmetric registered [ 3.244229] Asymmetric key parser 'x509' registered [ 3.245537] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.247558] io scheduler mq-deadline registered [ 3.248994] io scheduler kyber registered [ 3.250520] io scheduler bfq registered [ 3.252900] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.256248] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.259318] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.264111] ACPI: Power Button [PWRF] [ 3.270155] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.276569] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.299378] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.316152] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.336247] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.362519] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.388922] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.392960] Non-volatile memory driver v1.3 [ 3.393866] Linux agpgart interface v0.103 [ 3.422182] virtio_blk virtio1: [vda] 150000 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.423746] vda: detected capacity change from 0 to 76800000 [ 3.437767] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.440162] vdb: detected capacity change from 0 to 1073741824 [ 3.458197] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.461020] vdc: detected capacity change from 0 to 2621440000 [ 3.478033] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.481079] vdd: detected capacity change from 0 to 2621440000 [ 3.507775] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.510053] vde: detected capacity change from 0 to 4294967296 [ 3.530053] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.532021] vdf: detected capacity change from 0 to 4294967296 [ 3.537113] libphy: Fixed MDIO Bus: probed [ 3.545253] usbcore: registered new interface driver usbserial_generic [ 3.548207] usbserial: USB Serial support registered for generic [ 3.551200] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.556631] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.558445] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.561139] mousedev: PS/2 mouse device common for all mice [ 3.564237] rtc_cmos 00:05: RTC can wake from S4 [ 3.567442] rtc_cmos 00:05: registered as rtc0 [ 3.569854] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.571395] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.578178] intel_pstate: CPU model not supported [ 3.581710] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.585802] hid: raw HID events driver (C) Jiri Kosina [ 3.588052] usbcore: registered new interface driver usbhid [ 3.590295] usbhid: USB HID core driver [ 3.592431] drop_monitor: Initializing network drop monitor service [ 3.592523] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.594943] Initializing XFRM netlink socket [ 3.600566] NET: Registered protocol family 10 [ 3.603484] Segment Routing with IPv6 [ 3.605019] NET: Registered protocol family 17 [ 3.607221] mpls_gso: MPLS GSO support [ 3.612231] RAS: Correctable Errors collector initialized. [ 3.614600] AVX version of gcm_enc/dec engaged. [ 3.616392] AES CTR mode by8 optimization enabled [ 3.690290] sched_clock: Marking stable (3690268033, 0)->(4669313248, -979045215) [ 3.694329] registered taskstats version 1 [ 3.697103] Loading compiled-in X.509 certificates [ 3.699665] zswap: loaded using pool lzo/zbud [ 3.724623] Key type big_key registered [ 3.736827] Key type encrypted registered [ 3.738737] ima: No TPM chip found, activating TPM-bypass! [ 3.741138] ima: Allocated hash algorithm: sha1 [ 3.743552] ima: No architecture policies found [ 3.745898] evm: Initialising EVM extended attributes: [ 3.748286] evm: security.selinux [ 3.749779] evm: security.ima [ 3.751139] evm: security.capability [ 3.752647] evm: HMAC attrs: 0x1 [ 3.755259] rtc_cmos 00:05: setting system clock to 2026-09-17 08:50:25 UTC (1789635025) [ 3.762509] debug: unmapping init [mem 0xffffffff93003000-0xffffffff931fffff] [ 3.765987] debug: unmapping init [mem 0xffffffff91d82000-0xffffffff92058fff] [ 3.775152] Write protecting the kernel read-only data: 28672k [ 3.778916] debug: unmapping init [mem 0xffffffff90403000-0xffffffff905fffff] [ 3.782276] debug: unmapping init [mem 0xffffffff90d14000-0xffffffff90dfffff] [ 3.819187] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.831706] systemd[1]: Detected virtualization kvm. [ 3.833810] systemd[1]: Detected architecture x86-64. [ 3.835882] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.863878] systemd[1]: No hostname configured. [ 3.865764] systemd[1]: Set hostname to . [ 3.867983] random: systemd: uninitialized urandom read (16 bytes read) [ 3.871088] systemd[1]: Initializing machine ID from random generator. [ 3.919570] random: ln: uninitialized urandom read (6 bytes read) [ 4.012754] random: systemd: uninitialized urandom read (16 bytes read) [ 4.015796] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.020551] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.024429] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.545201] device-mapper: uevent: version 1.0.3 [ 4.546664] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.159842] random: fast init done [ 5.163775] virtio_net virtio0 ens2: renamed from eth0 [ 5.233302] scsi host0: ata_piix [ 5.259516] scsi host1: ata_piix [ 5.261157] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.263290] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.842894] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.261159] random: crng init done [ 10.262489] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.540763] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.648474] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.914605] SELinux: Disabled at runtime. [ 11.973268] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.981924] systemd[1]: Detected virtualization kvm. [ 11.983815] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.511455] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.514381] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.519221] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.526261] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.530118] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.536901] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.541821] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.615909] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.984161] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.268648] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.276905] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.406259] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.421401] EDAC sbridge: Ver: 1.1.2 [ 15.134995] Key type dns_resolver registered [ 15.463692] NFS: Registering the id_resolver key type [ 15.465353] Key type id_resolver registered [ 15.466534] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg202-server login: [ 51.646873] libcfs: loading out-of-tree module taints kernel. [ 51.726378] Key type ._llcrypt registered [ 51.730378] Key type .llcrypt registered [ 51.842835] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 70.055431] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 71.460150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 71.486694] alg: No test for adler32 (adler32-zlib) [ 73.128769] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 74.241846] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 76.152219] Key type lgssc registered [ 77.669792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 92.952933] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 94.949012] hrtimer: interrupt took 1966020 ns [ 130.457511] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 143.449215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 143.506535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.805684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 144.884693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 145.029199] Lustre: lustre-MDT0000: new disk, initializing [ 145.091524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 145.121109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 149.112215] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.343453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.453633] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 161.485957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 161.489102] Lustre: Skipped 1 previous similar message [ 161.581499] Lustre: lustre-MDT0001: new disk, initializing [ 161.642421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 161.699586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 161.713220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 166.362726] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.508530] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 180.957443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 181.318542] Lustre: lustre-OST0000: new disk, initializing [ 181.323762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 181.335146] Lustre: 8458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 181.427895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.538125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 183.544826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 183.651537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 188.134036] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.121143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.317753] Lustre: lustre-OST0001: new disk, initializing [ 205.322668] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 205.326250] Lustre: 9532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.447914] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.870119] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.177026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 214.206071] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 214.397606] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 227.068830] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 236.090440] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 244.309629] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.491778] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing yml_node [ 256.666398] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 259.562264] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 262.126353] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 263.487516] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 17 04:54:43 EDT 2026 [ 283.188654] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 285.253663] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 305.123114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 305.125862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.151391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.657092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.659670] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.946751] Lustre: server umount lustre-MDT0000 complete [ 313.486643] LustreError: 7478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635335 with bad export cookie 11944371369659208581 [ 313.487728] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.510460] LustreError: 7478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 316.905541] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.905660] LustreError: 10268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.906024] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.906027] Lustre: Skipped 3 previous similar messages [ 316.919562] Lustre: Skipped 3 previous similar messages [ 316.934051] LustreError: 10268:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 318.944901] LustreError: 6526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.944992] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.956583] LustreError: 6526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 318.959098] Lustre: Skipped 2 previous similar messages [ 320.047257] Lustre: server umount lustre-MDT0001 complete [ 323.957520] Lustre: server umount lustre-OST0000 complete [ 328.681570] Lustre: server umount lustre-OST0001 complete [ 337.129350] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 349.139529] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 402.943102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.089818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 403.111411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 403.155566] Lustre: lustre-MDT0000: new disk, initializing [ 403.222652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.242444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 407.531653] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 423.606976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.769650] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 423.817496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 423.823590] Lustre: Skipped 1 previous similar message [ 423.885371] Lustre: lustre-MDT0001: new disk, initializing [ 423.992497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 424.050115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 424.073246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 430.285906] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 444.884396] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 452.854634] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 462.291879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.761494] Lustre: lustre-OST0000: new disk, initializing [ 462.768171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.777867] Lustre: 17060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 462.901991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.495634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 464.506473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 464.688586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 471.095784] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 481.640542] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 489.819867] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 490.382543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 496.434524] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 496.860898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 500.706109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 500.720606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.722237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.734372] Lustre: Skipped 1 previous similar message [ 505.524481] Lustre: server umount lustre-OST0000 complete [ 510.955036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.955328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 510.957971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 510.957979] Lustre: Skipped 1 previous similar message [ 510.988507] Lustre: Skipped 1 previous similar message [ 516.032384] Lustre: server umount lustre-MDT0000 complete [ 516.067406] LustreError: 15100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.806931] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635542 with bad export cookie 11944371369659210114 [ 520.819530] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 520.821030] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.391216] Lustre: server umount lustre-MDT0001 complete [ 526.455635] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 04:59:05 (1789635545) [ 534.510472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.142862] LustreError: 19324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.159320] LustreError: 19324:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 535.219964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 540.582465] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 540.643510] LustreError: 19325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.860469] LustreError: 19324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.869234] LustreError: 19324:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 550.597315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.484409] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 566.575368] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 574.490682] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 582.829177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 590.723035] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 599.451805] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 615.416784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 615.436718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.452645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 615.455147] Lustre: Skipped 1 previous similar message [ 620.057729] Lustre: server umount lustre-OST0000 complete [ 630.357749] Lustre: server umount lustre-MDT0000 complete [ 630.757401] LustreError: 19325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.767659] LustreError: 19325:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 634.635912] LustreError: 19306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635656 with bad export cookie 11944371369659210989 [ 634.645125] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 634.650051] LustreError: 19306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 635.168459] Lustre: server umount lustre-MDT0001 complete [ 649.131624] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 652.863562] Key type lgssc unregistered [ 653.255816] LNet: 22893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.267661] LNetError: 22893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.324747] LNet: Removed LNI 192.168.202.102@tcp [ 655.746196] Key type .llcrypt unregistered [ 655.749072] Key type ._llcrypt unregistered [ 682.246996] Key type ._llcrypt registered [ 682.249921] Key type .llcrypt registered [ 682.394144] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 683.832260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 683.845176] alg: No test for adler32 (adler32-zlib) [ 684.916200] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 685.343461] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 687.040304] Key type lgssc registered [ 688.531734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 696.676851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 696.698835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.137923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 702.831934] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 707.556616] LustreError: 24303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.589705] LustreError: 24303:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 710.689905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.046678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 714.896430] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 722.963329] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 728.691635] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 735.685491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 736.139762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 742.240790] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 742.337161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 749.741129] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 765.926606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 765.928804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.958541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 765.974799] Lustre: Skipped 1 previous similar message [ 765.982935] Lustre: Skipped 1 previous similar message [ 769.507393] Lustre: server umount lustre-OST0000 complete [ 776.161550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.174917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 776.182769] Lustre: Skipped 1 previous similar message [ 776.201194] Lustre: Skipped 1 previous similar message [ 779.404731] Lustre: server umount lustre-MDT0000 complete [ 781.291245] LustreError: 25031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 781.311359] LustreError: 25031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 782.985710] LustreError: 27238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635804 with bad export cookie 16326169351892624921 [ 782.986907] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 782.996434] LustreError: 27238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 783.351791] Lustre: server umount lustre-MDT0001 complete [ 795.017117] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 797.900555] Key type lgssc unregistered [ 798.243659] LNet: 28277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.259326] LNetError: 28277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.288122] LNet: Removed LNI 192.168.202.102@tcp [ 799.244591] Key type .llcrypt unregistered [ 799.248556] Key type ._llcrypt unregistered [ 819.529275] Key type ._llcrypt registered [ 819.531762] Key type .llcrypt registered [ 819.634459] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 820.745950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 820.794100] alg: No test for adler32 (adler32-zlib) [ 821.854484] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 822.053647] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 823.712220] Key type lgssc registered [ 824.750835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 831.044524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 831.067683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.393368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 835.902803] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 842.722424] LustreError: 29665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 842.748107] LustreError: 29665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 842.821513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 843.145190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 846.465226] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 852.551213] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 858.265781] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 871.063204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 871.503770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 876.731416] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 877.688092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 884.337567] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 898.023052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 898.026962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 898.033197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 898.542209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 898.559374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 898.561074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 902.612937] Lustre: server umount lustre-OST0000 complete [ 908.774054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.783871] Lustre: Skipped 1 previous similar message [ 908.788707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 908.794612] Lustre: Skipped 1 previous similar message [ 912.679215] Lustre: server umount lustre-MDT0000 complete [ 913.891902] LustreError: 29666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.911237] LustreError: 29666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 916.107784] LustreError: 29648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635937 with bad export cookie 12230168331354251937 [ 916.109863] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 916.124188] LustreError: 29648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 916.452872] Lustre: server umount lustre-MDT0001 complete [ 927.526245] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 930.136966] Key type lgssc unregistered [ 930.461810] LNet: 33558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.468588] LNetError: 33558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.499909] LNet: Removed LNI 192.168.202.102@tcp [ 932.555066] Key type .llcrypt unregistered [ 932.560106] Key type ._llcrypt unregistered [ 954.980333] Key type ._llcrypt registered [ 954.981988] Key type .llcrypt registered [ 955.087020] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 956.253709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 956.273380] alg: No test for adler32 (adler32-zlib) [ 957.301142] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 957.489614] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 959.128773] Key type lgssc registered [ 960.111916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 966.996431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 967.038036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.606458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 973.386119] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 978.930493] LustreError: 34967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.954441] LustreError: 34967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 981.004514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.460400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 985.039888] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 992.571534] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 998.044577] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1004.431768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1004.712822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.455873] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1013.965067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1018.120855] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1027.553983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1027.566063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.574380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1028.076419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1028.081729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.105776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1029.609754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1032.947921] Lustre: server umount lustre-OST0000 complete [ 1037.794059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1037.804459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.814070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.818478] Lustre: Skipped 1 previous similar message [ 1039.845789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.505373] Lustre: server umount lustre-MDT0000 complete [ 1044.973360] LustreError: 34968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1044.996090] LustreError: 34968:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1045.576521] LustreError: 34948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636067 with bad export cookie 3359956237100451319 [ 1045.588728] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1045.592298] LustreError: 34948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1045.938196] Lustre: server umount lustre-MDT0001 complete [ 1055.270479] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1057.730532] Key type lgssc unregistered [ 1057.948031] LNet: 38497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.957925] LNetError: 38497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.971562] LNet: Removed LNI 192.168.202.102@tcp [ 1058.915108] Key type .llcrypt unregistered [ 1058.918410] Key type ._llcrypt unregistered [ 1080.827903] Key type ._llcrypt registered [ 1080.829586] Key type .llcrypt registered [ 1080.912372] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1081.770949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1081.802801] alg: No test for adler32 (adler32-zlib) [ 1082.865246] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1083.064523] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1084.720581] Key type lgssc registered [ 1085.796895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1092.459361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1092.478942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.844194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1097.762071] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1103.345328] LustreError: 39906:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.359625] LustreError: 39906:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1105.621548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1105.914191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1109.279899] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1115.219767] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1119.715307] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1125.001045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1125.259558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.743383] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1135.516710] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1138.600741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1145.829206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1145.832838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.849821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.570879] Lustre: lustre-OST0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1148.901421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1148.901930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.647913] Lustre: lustre-OST0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1151.654180] Lustre: Skipped 1 previous similar message [ 1156.754666] Lustre: lustre-OST0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1156.764363] Lustre: Skipped 2 previous similar messages [ 1159.139163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1159.266575] Lustre: server umount lustre-OST0000 complete [ 1177.569879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1177.575168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1177.582912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1177.586414] Lustre: Skipped 2 previous similar messages [ 1179.621059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.671309] Lustre: server umount lustre-MDT0000 complete [ 1184.745250] LustreError: 40633:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.756883] LustreError: 40633:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1185.809353] LustreError: 39886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636207 with bad export cookie 202850702055523522 [ 1185.810336] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1185.816786] LustreError: 39886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1192.185926] Lustre: server umount lustre-MDT0001 complete [ 1200.423355] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1202.900175] Key type lgssc unregistered [ 1203.169855] LNet: 43613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.187267] LNetError: 43613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.202994] LNet: Removed LNI 192.168.202.102@tcp [ 1203.893330] Key type .llcrypt unregistered [ 1203.895659] Key type ._llcrypt unregistered [ 1220.757199] Key type ._llcrypt registered [ 1220.758494] Key type .llcrypt registered [ 1220.839261] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1221.531755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1221.594619] alg: No test for adler32 (adler32-zlib) [ 1222.547811] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1222.763177] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1224.384372] Key type lgssc registered [ 1225.178336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1231.273534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1231.283547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.600631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.312866] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1242.745725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1243.013042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1246.155160] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1252.391842] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1257.115584] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1261.967311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1262.210102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.033484] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1273.187980] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1278.584965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1284.071036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.074970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1284.083893] Lustre: Skipped 2 previous similar messages [ 1284.100239] Lustre: Skipped 2 previous similar messages [ 1285.306829] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1289.191931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1289.201540] Lustre: Skipped 2 previous similar messages [ 1289.835486] Lustre: server umount lustre-MDT0000 complete [ 1290.389189] LustreError: 47204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.399359] LustreError: 47204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1293.063932] LustreError: 44980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636314 with bad export cookie 15902430453245685142 [ 1293.074779] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1294.308711] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.309386] LustreError: 44994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.323405] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1295.509817] LustreError: 47204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.337367] LustreError: 44994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.338529] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1298.371353] Lustre: Skipped 1 previous similar message [ 1299.476285] Lustre: server umount lustre-MDT0001 complete [ 1319.735843] Lustre: server umount lustre-OST0000 complete [ 1332.532872] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1334.600619] Key type lgssc unregistered [ 1334.848200] LNet: 48858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1334.861590] LNetError: 48858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.898379] LNet: Removed LNI 192.168.202.102@tcp [ 1335.586516] Key type .llcrypt unregistered [ 1335.590037] Key type ._llcrypt unregistered [ 1354.705353] Key type ._llcrypt registered [ 1354.709167] Key type .llcrypt registered [ 1354.803039] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1355.745943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1355.768809] alg: No test for adler32 (adler32-zlib) [ 1356.783629] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1356.950418] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1358.600213] Key type lgssc registered [ 1359.314507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1365.436674] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1365.452817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1381.024516] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1386.144759] LustreError: 50221:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 1412.769155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.837420] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1424.125724] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1589.990363] Lustre: server umount lustre-OST0000 complete [ 1602.210133] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1604.309878] Key type lgssc unregistered [ 1604.494577] LNet: 51897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.497645] LNetError: 51897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1604.506610] LNet: Removed LNI 192.168.202.102@tcp [ 1605.158828] Key type .llcrypt unregistered [ 1605.163691] Key type ._llcrypt unregistered [ 1624.878233] Key type ._llcrypt registered [ 1624.879976] Key type .llcrypt registered [ 1624.925387] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1625.697528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1625.787929] alg: No test for adler32 (adler32-zlib) [ 1626.822914] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1627.010312] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1628.672223] Key type lgssc registered [ 1629.664780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1636.576671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1636.599826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.082878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.022588] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1647.587721] LustreError: 53287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1647.616069] LustreError: 53287:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1649.038952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1649.274668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1652.794961] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1659.174680] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1664.543407] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1671.040210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1671.464786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.752384] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1682.742235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1683.458252] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1688.038295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.043592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1688.062973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1688.066145] Lustre: Skipped 1 previous similar message [ 1691.877306] Lustre: server umount lustre-OST0000 complete [ 1698.279099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.287265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1698.293486] Lustre: Skipped 2 previous similar messages [ 1698.306207] Lustre: Skipped 1 previous similar message [ 1701.532907] Lustre: server umount lustre-MDT0000 complete [ 1703.393644] LustreError: 53281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1703.405035] LustreError: 53281:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1704.703817] LustreError: 53266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636726 with bad export cookie 17063277817215969332 [ 1704.715682] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1704.717490] LustreError: 53266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1705.080666] Lustre: server umount lustre-MDT0001 complete [ 1715.891716] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1718.345594] Key type lgssc unregistered [ 1718.588437] LNet: 56815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.595503] LNetError: 56815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.656839] LNet: Removed LNI 192.168.202.102@tcp [ 1720.483864] Key type .llcrypt unregistered [ 1720.485245] Key type ._llcrypt unregistered [ 1738.664724] Key type ._llcrypt registered [ 1738.666956] Key type .llcrypt registered [ 1738.752733] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1739.518787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1739.562564] alg: No test for adler32 (adler32-zlib) [ 1740.572491] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1740.776842] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1742.440463] Key type lgssc registered [ 1743.359747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1750.342883] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1750.357732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1765.856882] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1770.977352] LustreError: 58177:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 1797.664346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.217675] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1809.210572] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1814.164570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.573170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.619932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1817.929115] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1819.625559] LustreError: 58202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.638276] LustreError: 58202:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1824.438798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.663987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1827.796654] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1834.057886] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1838.320038] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1840.101212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1840.102873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.107386] LustreError: Skipped 1 previous similar message [ 1840.122184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1840.133375] Lustre: Skipped 1 previous similar message [ 1843.175480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1843.184588] Lustre: Skipped 1 previous similar message [ 1846.255379] Lustre: server umount lustre-OST0000 complete [ 1866.012572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.022825] Lustre: Skipped 1 previous similar message [ 1866.025937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1869.026383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1869.029094] Lustre: Skipped 2 previous similar messages [ 1872.395286] Lustre: server umount lustre-MDT0000 complete [ 1875.428759] LustreError: 59215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1875.438868] LustreError: 59215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1875.747616] LustreError: 58184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636897 with bad export cookie 16856334958041851134 [ 1875.748670] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1875.757146] LustreError: 58184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1890.273407] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1890.524241] Lustre: server umount lustre-MDT0001 complete [ 1898.547541] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1900.599579] Key type lgssc unregistered [ 1900.854163] LNet: 61919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.868590] LNetError: 61919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.926205] LNet: Removed LNI 192.168.202.102@tcp [ 1902.759123] Key type .llcrypt unregistered [ 1902.761232] Key type ._llcrypt unregistered [ 1922.343852] Key type ._llcrypt registered [ 1922.347180] Key type .llcrypt registered [ 1922.454397] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1923.576219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1923.602496] alg: No test for adler32 (adler32-zlib) [ 1924.648542] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1924.861433] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1926.528191] Key type lgssc registered [ 1927.511895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1934.184457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1934.203672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.520628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.286686] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1946.089586] LustreError: 63308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.120801] LustreError: 63308:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1946.533448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.751350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1950.372933] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1956.806242] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1961.989617] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1967.340385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1967.623634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.282972] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1978.015797] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1980.915439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1995.174468] Lustre: server umount lustre-OST0000 complete [ 2001.378552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2001.380425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2001.395066] Lustre: Skipped 1 previous similar message [ 2001.401488] Lustre: Skipped 1 previous similar message [ 2006.500536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.609213] Lustre: server umount lustre-MDT0000 complete [ 2011.147779] LustreError: 63288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637032 with bad export cookie 12511333349506584994 [ 2011.180750] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2020.422122] Lustre: server umount lustre-MDT0001 complete [ 2029.992343] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2032.432931] Key type lgssc unregistered [ 2032.716662] LNet: 66864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.723994] LNetError: 66864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.734313] LNet: Removed LNI 192.168.202.102@tcp [ 2033.549482] Key type .llcrypt unregistered [ 2033.550720] Key type ._llcrypt unregistered [ 2052.680416] Key type ._llcrypt registered [ 2052.682742] Key type .llcrypt registered [ 2052.789625] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2053.609119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2053.625879] alg: No test for adler32 (adler32-zlib) [ 2054.598920] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2054.806193] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2056.448277] Key type lgssc registered [ 2057.300387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2064.286172] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 05:24:44 (1789637084) [ 2069.505578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2069.522895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.803431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2074.226459] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2080.382985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.635083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2084.874621] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2090.737606] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2094.877344] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2099.817826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2100.019341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.064939] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2109.726579] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2116.412769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2124.740131] Lustre: Failing over lustre-MDT0000 [ 2124.905414] Lustre: server umount lustre-MDT0000 complete [ 2126.508942] LustreError: 68539:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.530290] LustreError: 68539:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2126.822221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2126.825258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2126.836934] Lustre: Skipped 1 previous similar message [ 2131.223049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.311915] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2131.382933] Lustre: *** cfs_fail_loc=135, val=0*** [ 2131.458597] LustreError: 71648:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2131.466325] LustreError: 71648:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.102@tcp: cfg command failed: rc = -2 [ 2131.470615] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2131.470615] [ 2131.475234] LustreError: MGC192.168.202.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2131.485545] LustreError: 71637:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2131.491122] LustreError: 71637:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2131.496994] LustreError: 71637:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2131.501877] LustreError: 71637:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2131.513542] Lustre: server umount lustre-MDT0000 complete [ 2131.514857] LustreError: 71637:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2131.604456] LustreError: 68537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.626528] LustreError: 68537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2136.686132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.724245] LustreError: 68538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.735887] LustreError: 68538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2136.822370] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2137.029345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.075060] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2140.413297] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2141.842181] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2142.220668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2142.239865] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2142.286251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2147.298784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2147.299174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.309846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2147.331173] Lustre: Skipped 2 previous similar messages [ 2148.426032] Lustre: server umount lustre-OST0000 complete [ 2150.776377] Lustre: server umount lustre-MDT0000 complete [ 2152.418373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.419350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2152.425399] LustreError: 68537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.447022] LustreError: 68537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2153.215217] LustreError: 70383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637174 with bad export cookie 9644513623289393559 [ 2153.219720] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.222106] LustreError: 70383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2153.460532] Lustre: server umount lustre-MDT0001 complete [ 2161.262786] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2163.289896] Key type lgssc unregistered [ 2163.514789] LNet: 73421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2163.525244] LNetError: 73421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2163.551701] LNet: Removed LNI 192.168.202.102@tcp [ 2164.201882] Key type .llcrypt unregistered [ 2164.204202] Key type ._llcrypt unregistered [ 2181.089320] Key type ._llcrypt registered [ 2181.091944] Key type .llcrypt registered [ 2181.172513] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2182.007568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.021846] alg: No test for adler32 (adler32-zlib) [ 2183.076405] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2183.247968] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2184.984341] Key type lgssc registered [ 2186.076409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2192.061374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2192.076087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.384409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.481555] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2203.158612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.471619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2207.228666] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2213.335640] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2216.693408] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2220.264791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2220.472517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.662536] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2227.985795] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2239.928717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2247.950078] Lustre: Failing over lustre-MDT0000 [ 2248.138686] Lustre: server umount lustre-MDT0000 complete [ 2249.697311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2249.704100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.710622] LustreError: 74809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.717411] LustreError: 74809:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2250.211095] LustreError: 74810:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.214788] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.220457] LustreError: 74810:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2253.423199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.525380] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2253.630290] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2253.714948] LustreError: 77915:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2253.721292] LustreError: 77915:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.102@tcp: cfg command failed: rc = -9 [ 2253.725842] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2253.725842] [ 2253.730930] LustreError: MGC192.168.202.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2253.735939] LustreError: 77903:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2253.739170] LustreError: 77903:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2253.745309] LustreError: 77903:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2253.749049] LustreError: 77903:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2253.763865] Lustre: server umount lustre-MDT0000 complete [ 2253.766190] LustreError: 77903:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2255.000743] LustreError: 75211:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.136719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.215055] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.364556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.400131] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2260.118863] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2261.266783] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2263.537614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2263.567261] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2263.597663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2268.643116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.645169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2268.648836] Lustre: Skipped 2 previous similar messages [ 2268.657352] Lustre: Skipped 1 previous similar message [ 2272.509131] Lustre: server umount lustre-OST0000 complete [ 2278.882419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2278.883077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.888762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2278.900617] Lustre: Skipped 1 previous similar message [ 2278.909318] Lustre: Skipped 1 previous similar message [ 2281.264900] Lustre: server umount lustre-MDT0000 complete [ 2283.820606] LustreError: 74789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637305 with bad export cookie 8860888410260679647 [ 2283.823977] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2283.834430] LustreError: 74789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2284.175277] Lustre: server umount lustre-MDT0001 complete [ 2291.505850] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2293.310116] Key type lgssc unregistered [ 2293.478765] LNet: 79693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.482239] LNetError: 79693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.495293] LNet: Removed LNI 192.168.202.102@tcp [ 2294.121728] Key type .llcrypt unregistered [ 2294.126228] Key type ._llcrypt unregistered [ 2302.453639] Key type ._llcrypt registered [ 2302.455631] Key type .llcrypt registered [ 2302.524794] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2302.721454] Key type .llcrypt unregistered [ 2302.723467] Key type ._llcrypt unregistered [ 2321.108188] Key type ._llcrypt registered [ 2321.111554] Key type .llcrypt registered [ 2321.239375] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2322.460784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2322.666078] alg: No test for adler32 (adler32-zlib) [ 2323.705134] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2323.953265] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2325.640646] Key type lgssc registered [ 2327.179987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2334.098512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2334.116507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2335.410666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.663670] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2345.009509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.219118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2348.510252] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2353.817297] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2357.954897] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2363.170885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2363.396571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.496662] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2379.225833] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2381.803935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2395.105084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2395.119556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2395.128056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2397.157572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2397.160936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2397.162935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.169542] Lustre: Skipped 1 previous similar message [ 2399.262337] Lustre: server umount lustre-OST0000 complete [ 2402.276497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.289909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2402.296056] Lustre: Skipped 1 previous similar message [ 2407.394450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2407.396891] Lustre: Skipped 1 previous similar message [ 2408.443164] Lustre: server umount lustre-MDT0000 complete [ 2411.407659] LustreError: 82300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637433 with bad export cookie 1692799871347873551 [ 2411.411346] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2411.423696] LustreError: 82300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2411.760131] Lustre: server umount lustre-MDT0001 complete [ 2419.676775] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2421.498735] Key type lgssc unregistered [ 2421.757991] LNet: 85116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.762853] LNetError: 85116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.778287] LNet: Removed LNI 192.168.202.102@tcp [ 2422.411297] Key type .llcrypt unregistered [ 2422.413381] Key type ._llcrypt unregistered [ 2439.888777] Key type ._llcrypt registered [ 2439.890923] Key type .llcrypt registered [ 2439.995542] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2440.880525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2440.945611] alg: No test for adler32 (adler32-zlib) [ 2441.948264] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2442.155644] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2443.797166] Key type lgssc registered [ 2444.641189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2449.676682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2449.693405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.944636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2453.915594] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2459.964555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.232658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2463.380412] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2469.012678] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2473.682930] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2478.845839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2479.096897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.203719] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2488.607303] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2496.497432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2505.697883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2505.709154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.720401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2506.724785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2506.728774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2506.734500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.351564] Lustre: server umount lustre-OST0000 complete [ 2516.450831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2516.459240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.473636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2516.476774] Lustre: Skipped 1 previous similar message [ 2519.181740] Lustre: server umount lustre-MDT0000 complete [ 2522.093351] LustreError: 86502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.107763] LustreError: 86502:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2522.314272] LustreError: 86483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637544 with bad export cookie 14404314241517496731 [ 2522.318805] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2522.320560] LustreError: 86483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2522.566850] Lustre: server umount lustre-MDT0001 complete [ 2531.494974] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2533.790935] Key type lgssc unregistered [ 2534.041164] LNet: 90053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.053107] LNetError: 90053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2535.083467] LNet: Removed LNI 192.168.202.102@tcp [ 2536.132706] Key type .llcrypt unregistered [ 2536.138477] Key type ._llcrypt unregistered [ 2554.888552] Key type ._llcrypt registered [ 2554.890532] Key type .llcrypt registered [ 2554.966959] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2555.946462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2555.955655] alg: No test for adler32 (adler32-zlib) [ 2556.897542] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2557.013874] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2558.625791] Key type lgssc registered [ 2559.247989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2563.901672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2563.910055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.115344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2566.935049] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2570.945797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.127152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2573.341400] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2578.047618] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2581.611473] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2585.778150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2586.020207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.821616] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2590.119348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2594.238600] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2605.538986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2605.543134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.550627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2606.561925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2606.563892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2606.565320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.287935] Lustre: server umount lustre-OST0000 complete [ 2611.685636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.686832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2611.704582] Lustre: Skipped 1 previous similar message [ 2611.710603] Lustre: Skipped 2 previous similar messages [ 2616.304443] Lustre: server umount lustre-MDT0000 complete [ 2616.804040] LustreError: 91459:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2616.811947] LustreError: 91459:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2618.884491] LustreError: 91420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637640 with bad export cookie 1885481450496244598 [ 2618.892196] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2618.893829] LustreError: 91420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2619.153408] Lustre: server umount lustre-MDT0001 complete [ 2626.490899] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2628.072852] Key type lgssc unregistered [ 2628.233497] LNet: 94971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2628.240771] LNetError: 94971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2628.250817] LNet: Removed LNI 192.168.202.102@tcp [ 2628.751809] Key type .llcrypt unregistered [ 2628.753397] Key type ._llcrypt unregistered [ 2642.872177] Key type ._llcrypt registered [ 2642.873661] Key type .llcrypt registered [ 2642.918721] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2643.382409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2643.417579] alg: No test for adler32 (adler32-zlib) [ 2644.282855] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2644.392766] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2646.000200] Key type lgssc registered [ 2647.103044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2655.118411] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2655.132982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.689640] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2675.808676] LustreError: 96332:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 2702.560433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.861610] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2721.319312] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2731.765629] Lustre: server umount lustre-OST0000 complete [ 2744.160024] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 05:36:03 (1789637763) [ 2754.244477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.718230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2759.457475] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2760.164761] LustreError: 97974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.181113] LustreError: 97974:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2764.276930] LustreError: 97975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.378763] LustreError: 97974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.515394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.854819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2775.346402] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2783.883349] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2790.278514] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2797.569302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2797.941423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2800.022639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2804.623361] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2812.355311] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2829.810285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.842382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2829.853593] Lustre: Skipped 1 previous similar message [ 2832.893306] Lustre: server umount lustre-OST0000 complete [ 2840.036398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.044625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2840.083689] Lustre: Skipped 2 previous similar messages [ 2840.106945] Lustre: Skipped 1 previous similar message [ 2843.552648] Lustre: server umount lustre-MDT0000 complete [ 2845.156219] LustreError: 97974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2845.205799] LustreError: 97974:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2847.118343] LustreError: 100628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637868 with bad export cookie 3859254685324927049 [ 2847.136518] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2847.139967] LustreError: 100628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2847.440165] Lustre: server umount lustre-MDT0001 complete [ 2860.756286] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2864.569835] Key type lgssc unregistered [ 2864.725772] LNet: 101674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2864.728817] LNetError: 101674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2864.741513] LNet: Removed LNI 192.168.202.102@tcp [ 2865.637506] Key type .llcrypt unregistered [ 2865.640638] Key type ._llcrypt unregistered [ 2909.053045] Key type ._llcrypt registered [ 2909.055570] Key type .llcrypt registered [ 2909.190145] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2910.704614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2911.148155] alg: No test for adler32 (adler32-zlib) [ 2912.150881] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2912.348759] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2914.016452] Key type lgssc registered [ 2914.947644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2922.562297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2922.591432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2924.181150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.164616] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2933.730169] LustreError: 103860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.758414] LustreError: 103860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2938.859360] LustreError: 103859:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.738077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.220279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.795755] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2954.272742] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2960.963204] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2968.665764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2969.019646] LustreError: 105681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.333892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.708386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2978.770993] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2988.880254] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3000.303745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.313318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3000.329458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3000.335643] Lustre: Skipped 1 previous similar message [ 3004.675624] Lustre: server umount lustre-OST0000 complete [ 3010.533661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.540386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.555973] Lustre: Skipped 2 previous similar messages [ 3010.578881] Lustre: Skipped 1 previous similar message [ 3015.649381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.659533] Lustre: Skipped 1 previous similar message [ 3020.783313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.794218] Lustre: Skipped 1 previous similar message [ 3023.332328] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3023.569703] Lustre: server umount lustre-MDT0000 complete [ 3025.898512] LustreError: 103854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.921761] LustreError: 103854:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3027.725783] LustreError: 103840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789638049 with bad export cookie 1483206977094778425 [ 3027.735851] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3027.737698] LustreError: 103840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3028.148831] Lustre: server umount lustre-MDT0001 complete [ 3042.256723] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3045.669675] Key type lgssc unregistered [ 3046.076750] LNet: 107395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3046.088570] LNetError: 107395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3046.109615] LNet: Removed LNI 192.168.202.102@tcp [ 3047.067157] Key type .llcrypt unregistered [ 3047.072451] Key type ._llcrypt unregistered [ 3067.071323] Key type ._llcrypt registered [ 3067.077777] Key type .llcrypt registered [ 3067.204400] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3068.294733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3068.333473] alg: No test for adler32 (adler32-zlib) [ 3069.631016] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 3070.025280] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3071.808878] Key type lgssc registered [ 3073.066580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3081.998847] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3082.061714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3097.632733] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3102.754132] LustreError: 108635:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 3129.376275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3135.534447] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3145.036515] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3155.108694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.674761] LustreError: MGC192.168.202.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3155.682250] LustreError: 109661:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3155.687925] LustreError: 109661:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3155.694910] LustreError: 109661:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3155.834813] Lustre: server umount lustre-MDT0000 complete [ 3155.836806] LustreError: 109661:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3172.320160] Lustre: 107971:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789638178/real 1789638178] req@000000007d442bdb x1876571550911872/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789638194 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3172.347262] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.361119] LustreError: 108648:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3172.959669] Lustre: server umount lustre-OST0000 complete [ 3188.698422] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3203.715874] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3245.934367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.327209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3246.378211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3246.545278] Lustre: lustre-MDT0000: new disk, initializing [ 3246.685554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3246.703952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3252.247217] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3263.163321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.301614] Lustre: 112473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3263.401902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3263.408130] Lustre: Skipped 1 previous similar message [ 3263.545357] Lustre: lustre-MDT0001: new disk, initializing [ 3263.708707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3263.755509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3263.768941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3269.673692] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3283.672653] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3291.090806] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3299.246122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3299.645994] Lustre: lustre-OST0000: new disk, initializing [ 3299.656390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3299.676051] Lustre: 114442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3299.785400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.620403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3307.658595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3307.823287] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3307.826482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3318.630917] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3322.986186] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3323.212421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3327.975311] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3328.213666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3333.601477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3333.606410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.622859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3333.627821] Lustre: Skipped 1 previous similar message [ 3333.645897] Lustre: Skipped 1 previous similar message [ 3336.430611] Lustre: server umount lustre-OST0000 complete [ 3343.853503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.870875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.884466] Lustre: Skipped 1 previous similar message [ 3343.935826] Lustre: Skipped 1 previous similar message [ 3347.508575] Lustre: server umount lustre-MDT0000 complete [ 3352.274096] LustreError: 112467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789638374 with bad export cookie 15854490268115210947 [ 3352.285423] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3352.292341] LustreError: 112467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3352.879452] Lustre: server umount lustre-MDT0001 complete [ 3362.623598] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 05:46:22 (1789638382) [ 3366.289226] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3382.954678] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3394.949010] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3436.603907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.891302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3436.923768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3437.005172] Lustre: lustre-MDT0000: new disk, initializing [ 3437.080037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3437.091962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3442.614424] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3454.789478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.877860] Lustre: 119346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3454.924829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3454.932378] Lustre: Skipped 1 previous similar message [ 3454.986938] Lustre: lustre-MDT0001: new disk, initializing [ 3455.109568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3455.148648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3455.164967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3459.429277] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3472.879637] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3481.066204] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3489.042474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3489.505896] Lustre: lustre-OST0000: new disk, initializing [ 3489.515738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3489.528926] Lustre: 121317:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3489.637201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.659603] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3498.146937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3498.164604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3498.345150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3510.171697] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3518.097991] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3518.456561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3524.269914] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3524.601777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3529.208554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3529.229594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.232500] LustreError: Skipped 2 previous similar messages [ 3529.233604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3529.262396] Lustre: Skipped 1 previous similar message [ 3534.311829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3534.324278] Lustre: Skipped 2 previous similar messages [ 3534.588243] Lustre: server umount lustre-OST0000 complete [ 3539.426490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3539.453897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3544.959464] Lustre: server umount lustre-MDT0000 complete [ 3549.378023] LustreError: 119338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789638571 with bad export cookie 15854490268115211829 [ 3549.384285] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3549.390453] LustreError: 119338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3549.744203] Lustre: server umount lustre-MDT0001 complete [ 3558.751222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.091557] LustreError: 123535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3559.113767] LustreError: 123535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3559.167328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3563.881373] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3564.514464] LustreError: 123536:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3568.623867] LustreError: 123535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3573.180245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.396784] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3586.632396] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3592.419089] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3600.423693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3607.887881] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3617.146729] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3625.726508] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3633.636793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3633.642270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3633.653644] Lustre: Skipped 1 previous similar message [ 3633.664522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3633.698367] Lustre: Skipped 4 previous similar messages [ 3638.659725] Lustre: server umount lustre-OST0000 complete [ 3645.411497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3645.418618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.431901] Lustre: Skipped 1 previous similar message [ 3650.725131] Lustre: server umount lustre-MDT0000 complete [ 3654.114135] LustreError: 123536:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.131683] LustreError: 123536:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3655.634176] LustreError: 123517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789638677 with bad export cookie 15854490268115212711 [ 3655.644264] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3655.656985] LustreError: 123517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3656.053982] Lustre: server umount lustre-MDT0001 complete [ 3668.330053] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3671.898393] Key type lgssc unregistered [ 3672.224650] LNet: 127175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3672.230822] LNetError: 127175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3672.245229] LNet: Removed LNI 192.168.202.102@tcp [ 3673.377692] Key type .llcrypt unregistered [ 3673.379966] Key type ._llcrypt unregistered [ 3694.172748] Key type ._llcrypt registered [ 3694.175353] Key type .llcrypt registered [ 3694.290185] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3710.850715] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3712.448168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3712.461804] alg: No test for adler32 (adler32-zlib) [ 3713.623949] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 3713.922540] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3715.592240] Key type lgssc registered [ 3716.967220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3757.198630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3757.231114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.613333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3758.661850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3758.840179] Lustre: lustre-MDT0000: new disk, initializing [ 3758.991243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3759.028905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3763.600902] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3774.611911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3774.754506] Lustre: 130330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3774.810428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3774.814969] Lustre: Skipped 1 previous similar message [ 3774.921916] Lustre: lustre-MDT0001: new disk, initializing [ 3775.101188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3775.173965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3775.177884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3780.371730] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3791.643642] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3797.946561] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3805.660588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3805.903407] Lustre: lustre-OST0000: new disk, initializing [ 3805.906580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3805.911356] Lustre: 132299:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3805.992141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3812.458456] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3812.936476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3812.946275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3813.116074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3821.739599] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3827.548748] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3827.923142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3834.102456] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3834.465946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3838.946278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3838.957967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3838.983939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3839.002563] Lustre: Skipped 1 previous similar message [ 3844.070289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3844.076506] Lustre: Skipped 1 previous similar message [ 3849.190451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3849.203418] Lustre: Skipped 1 previous similar message [ 3851.232669] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3851.481145] Lustre: server umount lustre-OST0000 complete [ 3856.866097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3856.871389] LustreError: Skipped 1 previous similar message [ 3856.874048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3856.882143] Lustre: Skipped 1 previous similar message [ 3856.885987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3859.430165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.336681] Lustre: server umount lustre-MDT0000 complete [ 3865.125268] LustreError: 130322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789638886 with bad export cookie 10535982313309243277 [ 3865.128365] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3865.139944] LustreError: 130322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3865.389956] Lustre: server umount lustre-MDT0001 complete [ 3873.813977] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:54:53 (1789638893) [ 3881.494522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.114916] LustreError: 134712:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.135545] LustreError: 134712:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3882.188299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3886.497700] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3887.591296] LustreError: 134713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3891.684982] LustreError: 134712:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3895.907978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.238582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3901.137634] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3909.239310] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3915.125154] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3920.866796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3920.868769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3920.887274] Lustre: Skipped 1 previous similar message [ 3920.917107] Lustre: Skipped 3 previous similar messages [ 3931.106095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3931.110294] Lustre: Skipped 3 previous similar messages [ 3932.128340] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3932.594890] Lustre: server umount lustre-MDT0000 complete [ 3936.234430] LustreError: 135710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3936.266376] LustreError: 135710:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3937.097628] LustreError: 134695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789638958 with bad export cookie 10535982313309244159 [ 3937.097913] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3937.101755] LustreError: 134695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3937.354363] Lustre: server umount lustre-MDT0001 complete [ 3946.331348] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:56:06 (1789638966) [ 3954.306751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3969.953513] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3975.072509] LustreError: 137126:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 4000.738149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4008.335112] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4018.018347] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4020.319812] Lustre: server umount lustre-OST0000 complete [ 4029.054332] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:57:29 (1789639049) [ 4035.488120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.909823] LustreError: 138574:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.990930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.687528] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4045.283625] LustreError: 138575:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4045.292428] LustreError: 138575:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4047.329799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.373980] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4057.665216] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4062.916338] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4068.583084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4068.846088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.848598] Lustre: Skipped 1 previous similar message [ 4074.561130] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4081.083678] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4088.805485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4088.813946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.828182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.851913] Lustre: server umount lustre-MDT0000 complete [ 4096.486109] LustreError: 138575:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4096.499795] LustreError: 138575:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4097.109938] LustreError: 138555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789639118 with bad export cookie 10535982313309244824 [ 4097.117289] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4097.128981] LustreError: 138555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4097.453911] Lustre: server umount lustre-MDT0001 complete [ 4111.063196] Lustre: server umount lustre-OST0000 complete [ 4118.038775] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:58:58 (1789639138) [ 4122.911640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4126.907921] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4128.738436] LustreError: 142227:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4128.749174] LustreError: 142227:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4133.638437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.869993] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4133.872589] Lustre: Skipped 3 previous similar messages [ 4133.964637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4133.974630] Lustre: Skipped 1 previous similar message [ 4137.321779] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4143.470107] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4148.052622] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4153.404060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4158.480692] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4164.585430] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4167.693377] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4169.012749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4169.968672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4173.745311] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4173.958911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4179.427167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4179.429834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.433218] LustreError: Skipped 1 previous similar message [ 4179.442959] Lustre: Skipped 3 previous similar messages [ 4181.735803] Lustre: server umount lustre-OST0000 complete [ 4190.638473] Lustre: server umount lustre-MDT0000 complete [ 4193.197892] LustreError: 142207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789639214 with bad export cookie 10535982313309245895 [ 4193.205869] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4193.207241] LustreError: 142207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4193.379368] Lustre: server umount lustre-MDT0001 complete [ 4199.287654] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 06:00:19 (1789639219) [ 4203.735350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4219.232604] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4224.352432] LustreError: 146396:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 4254.182367] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4259.369569] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4264.188545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.505627] LustreError: 146421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.546197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4264.554336] Lustre: Skipped 2 previous similar messages [ 4264.591491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4267.428891] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4279.206777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.609362] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4297.671158] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4307.219641] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4313.226935] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4313.945769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4320.253983] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4320.392686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4325.873866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.880596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4325.886585] Lustre: Skipped 3 previous similar messages [ 4325.890577] Lustre: Skipped 6 previous similar messages [ 4325.897254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4330.290325] Lustre: server umount lustre-MDT0000 complete [ 4334.922540] LustreError: 148185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789639356 with bad export cookie 10535982313309246917 [ 4334.936678] LustreError: 148185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4334.937606] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4335.243327] Lustre: server umount lustre-MDT0001 complete [ 4348.598994] Lustre: server umount lustre-OST0000 complete [ 4360.821787] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 06:02:59 (1789639379) [ 4371.927200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4387.682161] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4392.801103] LustreError: 150604:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 4425.350633] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4432.721578] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4440.109228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.235544] LustreError: 128547:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005c109e29 x1876572226346112/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4440.609660] LustreError: 150629:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4440.629309] LustreError: 150629:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 4440.769988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4445.660862] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4454.308409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4459.295122] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4466.929127] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4472.411229] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4479.093235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4479.214604] Lustre: lustre-OST0001: new disk, initializing [ 4479.217241] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4479.221912] Lustre: 153475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4485.683813] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4489.318785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4489.328609] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4489.480053] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4495.731898] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4500.595951] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4500.770394] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4506.709240] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4506.970246] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4510.193974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4510.210746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4510.221872] Lustre: Skipped 2 previous similar messages [ 4510.235957] Lustre: Skipped 3 previous similar messages [ 4515.241611] Lustre: server umount lustre-OST0000 complete [ 4520.420027] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4520.430088] LustreError: Skipped 1 previous similar message [ 4533.216133] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4533.528518] Lustre: server umount lustre-OST0001 complete [ 4552.161060] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4552.856330] Lustre: server umount lustre-MDT0000 complete [ 4557.507421] LustreError: 150610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789639579 with bad export cookie 10535982313309247869 [ 4557.513471] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4557.528453] LustreError: 150610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4558.103250] Lustre: server umount lustre-MDT0001 complete [ 4582.564372] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 06:06:42 (1789639602) [ 4584.223168] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4586.673812] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4589.024253] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 06:06:48 (1789639608) [ 4599.597637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4600.237065] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4600.257547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4600.388603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4600.392815] Lustre: Skipped 5 previous similar messages [ 4607.913497] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4621.173856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.251809] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4621.275360] Lustre: 157042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4627.186904] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4635.885137] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4642.533222] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4650.017871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4650.303877] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4656.265640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4657.303117] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4665.364256] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4670.946053] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4671.221482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4676.456183] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4676.744775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4681.697409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4681.707742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4681.727860] Lustre: Skipped 4 previous similar messages [ 4685.102392] Lustre: server umount lustre-OST0000 complete [ 4708.481300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4715.577327] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4723.188229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4723.227743] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4725.971925] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4732.459340] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4732.483718] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4733.176200] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4733.497759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4740.821649] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4741.073859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4747.523489] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff90be198dc800.ost_server_uuid 50 [ 4748.982128] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff90be198dc800.ost_server_uuid in IDLE state after 0 sec [ 4749.160785] LustreError: 159612:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.2@tcp) returned error from blocking AST (req@0000000007bbc2b6 x1876572226459392 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000338b9641/0x923754178898a2d7 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.2@tcp remote: 0xc1bf8d73985fe9f5 expref: 5 pid: 157047 timeout: 4849 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4749.219269] LustreError: lustre-MDT0000: A client on nid 192.168.202.2@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4749.232601] LustreError: 157037:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.2@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000338b9641/0x923754178898a2d7 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.2@tcp remote: 0xc1bf8d73985fe9f5 expref: 6 pid: 157047 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4753.896334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4753.903459] LustreError: Skipped 1 previous similar message [ 4759.351914] Lustre: server umount lustre-OST0000 complete [ 4768.229859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4768.232286] Lustre: Skipped 20 previous similar messages [ 4768.933756] Lustre: server umount lustre-MDT0000 complete [ 4773.345588] LustreError: 158506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.373379] LustreError: 158506:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 4773.637764] LustreError: 157033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789639795 with bad export cookie 10535982313309249052 [ 4773.649562] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4773.656836] LustreError: 157033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4780.377902] Lustre: server umount lustre-MDT0001 complete [ 4793.225469] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 4796.163745] Key type lgssc unregistered [ 4796.574823] LNet: 163157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4796.588681] LNetError: 163157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4796.630226] LNet: Removed LNI 192.168.202.102@tcp [ 4797.966541] Key type .llcrypt unregistered [ 4797.970076] Key type ._llcrypt unregistered [ 4823.085698] Key type ._llcrypt registered [ 4823.089316] Key type .llcrypt registered [ 4823.257649] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4824.841060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4824.885610] alg: No test for adler32 (adler32-zlib) [ 4826.096676] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 4826.383674] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 4828.032280] Key type lgssc registered [ 4829.511902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4838.820526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4838.835552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.557272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4846.114266] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4851.174242] LustreError: 164568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.189794] LustreError: 164568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4855.263834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4855.681657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4860.558190] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4867.841480] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4874.156214] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4882.350723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4882.743119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4885.832862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4890.986022] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4901.140631] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4907.565733] Lustre: Failing over lustre-MDT0000 [ 4908.104585] Lustre: server umount lustre-MDT0000 complete [ 4908.235289] LustreError: 164565:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.588433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4911.592304] LustreError: 164563:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.610250] Lustre: Skipped 2 previous similar messages [ 4911.628334] LustreError: 164563:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4916.711986] LustreError: 165568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4916.731233] LustreError: 165568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4920.682554] LustreError: 164548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.2@tcp arrived at 1789639942 with bad export cookie 934790409768394824 [ 4920.693505] LustreError: 164548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4921.827719] LustreError: 164564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.856716] LustreError: 164564:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4927.976418] Lustre: 163880:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789639933/real 1789639933] req@0000000046503c0b x1876573393283712/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789639949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4928.006621] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4929.870189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4933.090374] LustreError: 164569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4933.108303] LustreError: 164569:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4938.233542] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcf90aecc9afe0aa [ 4938.248819] Lustre: MGC192.168.202.102@tcp: Connection restored to 0@lo (at 0@lo) [ 4938.516225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4938.580403] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4943.154349] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4943.849860] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4943.851606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4945.517828] Lustre: lustre-MDT0000: Denying connection for new client 4c98801a-059a-4b7e-9b9b-56f83abe862c (at 192.168.202.2@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 4950.676270] Lustre: lustre-MDT0000: Denying connection for new client 4c98801a-059a-4b7e-9b9b-56f83abe862c (at 192.168.202.2@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 4957.669368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4957.680258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4957.703322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4959.210703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4959.210892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4961.028240] Lustre: server umount lustre-OST0000 complete [ 4965.312567] LustreError: 168307:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4965.326860] Lustre: 167656:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4965.335946] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4965.359505] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4965.373637] LustreError: 167656:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e88525d8 x1876573393304576/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4965.399104] LustreError: 167656:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4965.409657] LustreError: 167656:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4965.419519] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 4965.836763] Lustre: server umount lustre-MDT0000 complete [ 4969.441161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.491665] LustreError: 164564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4969.508511] LustreError: 164564:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4970.503322] LustreError: 164548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789639992 with bad export cookie 934790409768394922 [ 4970.509902] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4970.530611] LustreError: 164548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4970.543588] LustreError: 163876:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4971.040150] Lustre: server umount lustre-MDT0001 complete [ 4984.753945] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 4988.109795] Key type lgssc unregistered [ 4988.466678] LNet: 169060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4988.476700] LNetError: 169060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4988.497224] LNet: Removed LNI 192.168.202.102@tcp [ 4989.542786] Key type .llcrypt unregistered [ 4989.545739] Key type ._llcrypt unregistered [ 5015.320161] Key type ._llcrypt registered [ 5015.321827] Key type .llcrypt registered [ 5015.410500] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5016.622754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5016.639072] alg: No test for adler32 (adler32-zlib) [ 5017.651838] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 5017.963566] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5019.720266] Key type lgssc registered [ 5021.222779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5030.715034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5030.737802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5032.299510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5037.027574] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5041.658416] LustreError: 170470:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5041.687698] LustreError: 170470:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5046.773547] LustreError: 170471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.736378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5048.234365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5052.952384] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5060.175524] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5065.326481] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5072.088743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5072.473832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5076.645117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5079.093789] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5087.323877] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5097.561084] Lustre: server umount lustre-OST0000 complete [ 5104.611390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5104.616437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5104.639497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5107.682044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5107.684337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5107.717849] Lustre: Skipped 1 previous similar message [ 5112.804489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5112.812089] Lustre: Skipped 1 previous similar message [ 5113.894749] Lustre: 169781:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789640119/real 1789640119] req@00000000de2a7598 x1876573593563008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789640135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5113.897106] Lustre: server umount lustre-MDT0000 complete [ 5113.923805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.928825] LustreError: 171210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.968068] LustreError: 171210:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5115.872174] Lustre: 169782:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789640121/real 1789640121] req@00000000314d1213 x1876573593563520/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789640137 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5117.396668] LustreError: 170450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640139 with bad export cookie 15440035434973830668 [ 5117.409854] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5117.410211] LustreError: 170450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5117.763649] Lustre: server umount lustre-MDT0001 complete [ 5127.608038] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5130.116772] Key type lgssc unregistered [ 5130.452477] LNet: 174002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5130.458139] LNetError: 174002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5130.471937] LNet: Removed LNI 192.168.202.102@tcp [ 5131.190614] Key type .llcrypt unregistered [ 5131.194668] Key type ._llcrypt unregistered [ 5164.000314] Key type ._llcrypt registered [ 5164.008058] Key type .llcrypt registered [ 5164.137781] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5165.132161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5165.329973] alg: No test for adler32 (adler32-zlib) [ 5166.419867] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 5166.714271] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5168.416373] Key type lgssc registered [ 5169.683505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5177.271181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5177.294721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5178.654586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5182.743289] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5189.097781] LustreError: 175700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5189.112305] LustreError: 175700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5190.707977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5191.128470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5195.237309] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5201.331747] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5206.255305] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5212.457409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5212.756923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.180648] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5224.053757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5225.088842] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5231.585238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5231.600839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.616032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5234.672071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5234.678413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5236.969027] Lustre: server umount lustre-OST0000 complete [ 5242.339943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5242.349907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5242.359503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5242.365766] Lustre: Skipped 1 previous similar message [ 5244.900911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5244.901355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5250.018849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5250.022284] Lustre: Skipped 2 previous similar messages [ 5255.139579] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5255.310827] Lustre: server umount lustre-MDT0000 complete [ 5259.010284] LustreError: 175681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640280 with bad export cookie 3778122116682122135 [ 5259.013725] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5259.023459] LustreError: 175681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5259.504280] Lustre: server umount lustre-MDT0001 complete [ 5270.542730] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5274.031773] Key type lgssc unregistered [ 5274.285857] LNet: 179233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5274.295270] LNetError: 179233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5275.376910] LNet: Removed LNI 192.168.202.102@tcp [ 5276.306364] Key type .llcrypt unregistered [ 5276.309071] Key type ._llcrypt unregistered [ 5301.266197] Key type ._llcrypt registered [ 5301.270226] Key type .llcrypt registered [ 5301.359785] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5302.198155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5302.304950] alg: No test for adler32 (adler32-zlib) [ 5303.298352] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 5303.528968] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5305.288747] Key type lgssc registered [ 5306.395793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5318.090685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5318.105317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5319.459183] Lustre: *** cfs_fail_loc=135, val=0*** [ 5319.527195] LustreError: 180755:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5319.531547] LustreError: 180755:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.102@tcp: cfg command failed: rc = -2 [ 5319.538136] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5319.538136] [ 5319.544965] LustreError: MGC192.168.202.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5319.554036] LustreError: 180716:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5319.561280] LustreError: 180716:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5319.567430] LustreError: 180716:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5319.571213] LustreError: 180716:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5319.577731] LustreError: 180716:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.102@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000bc3aa06d) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5319.645853] Lustre: server umount lustre-MDT0000 complete [ 5319.649463] LustreError: 180716:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5326.712918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5327.286714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5332.136893] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5336.554029] LustreError: 181127:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5336.580822] LustreError: 181127:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5341.609493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5341.666967] LustreError: 181128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5342.102872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5347.034883] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5355.616439] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5361.617422] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5363.895382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5363.896138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5363.901109] Lustre: Skipped 1 previous similar message [ 5363.918032] Lustre: Skipped 1 previous similar message [ 5367.013912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5370.310380] Lustre: server umount lustre-MDT0000 complete [ 5372.385181] LustreError: 181128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5372.403334] LustreError: 181128:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5373.829617] LustreError: 181108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640395 with bad export cookie 17707959627416958095 [ 5373.850506] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5374.133825] Lustre: server umount lustre-MDT0001 complete [ 5385.689328] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5388.365864] Key type lgssc unregistered [ 5388.641732] LNet: 183446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5388.665754] LNetError: 183446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5389.745971] LNet: Removed LNI 192.168.202.102@tcp [ 5390.698805] Key type .llcrypt unregistered [ 5390.701545] Key type ._llcrypt unregistered [ 5413.940482] Key type ._llcrypt registered [ 5413.943247] Key type .llcrypt registered [ 5414.019461] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5414.284323] Key type .llcrypt unregistered [ 5414.287118] Key type ._llcrypt unregistered [ 5436.711785] Key type ._llcrypt registered [ 5436.714107] Key type .llcrypt registered [ 5436.819815] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5437.734852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5437.772858] alg: No test for adler32 (adler32-zlib) [ 5438.811936] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 5439.075564] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5440.755685] Key type lgssc registered [ 5441.988677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5450.623851] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5450.655577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5466.208736] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5471.328357] LustreError: 185570:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 5498.017328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5505.147634] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5513.887415] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5521.269513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.855855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5521.954821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5527.019452] LustreError: 185596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5527.046259] LustreError: 185596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5527.111357] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5532.139813] LustreError: 185597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5533.164891] LustreError: 185596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5533.201545] LustreError: 185596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5537.091597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.441736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5541.907133] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5549.336993] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5556.250197] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5560.045676] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5573.089538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5573.095656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.103280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5573.601652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5573.605992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.613293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5577.579282] Lustre: server umount lustre-OST0000 complete [ 5583.335287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5583.359046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5583.395581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5588.761521] Lustre: server umount lustre-MDT0000 complete [ 5588.969597] LustreError: 186607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5589.005456] LustreError: 186607:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5594.089129] LustreError: 187144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5594.111204] LustreError: 187144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5594.172321] LustreError: 188233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640615 with bad export cookie 2444489194653800342 [ 5594.185307] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5594.188404] LustreError: 188233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5594.750982] Lustre: server umount lustre-MDT0001 complete [ 5606.022788] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5609.100627] Key type lgssc unregistered [ 5609.456709] LNet: 189536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5609.473071] LNetError: 189536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5609.520149] LNet: Removed LNI 192.168.202.102@tcp [ 5610.876260] Key type .llcrypt unregistered [ 5610.879091] Key type ._llcrypt unregistered [ 5640.055330] Key type ._llcrypt registered [ 5640.056924] Key type .llcrypt registered [ 5640.137039] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5641.170666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5641.186879] alg: No test for adler32 (adler32-zlib) [ 5642.644024] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 5642.997469] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5644.712500] Key type lgssc registered [ 5646.488945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5656.531769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5656.562115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5658.201470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5663.932549] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5667.302297] LustreError: 190949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.330495] LustreError: 190949:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5672.426292] LustreError: 190948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5674.605559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.032707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5681.352925] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5692.911802] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5700.704054] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5709.672805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5710.296322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5717.147729] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5726.358402] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5727.748265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5735.739790] Lustre: Failing over lustre-MDT0000 [ 5735.965418] Lustre: server umount lustre-MDT0000 complete [ 5736.420910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5736.439994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5736.475926] LustreError: 190949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5737.958845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.972357] Lustre: Skipped 1 previous similar message [ 5742.739235] LustreError: 192192:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.784684] LustreError: 192192:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 5747.869643] LustreError: 190943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5747.909864] LustreError: 190943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5753.320086] Lustre: 190260:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789640759/real 1789640759] req@00000000a3929d15 x1876574248924800/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789640775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5753.367638] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5756.034621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5758.123460] LustreError: 190945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.160922] LustreError: 190945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5765.036092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5765.103446] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5769.390775] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5770.234597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5770.287575] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5770.341425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5770.930413] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5777.008245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5785.106431] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5795.816326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5795.829544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5795.837893] Lustre: Skipped 1 previous similar message [ 5795.847348] Lustre: Skipped 1 previous similar message [ 5799.029517] Lustre: server umount lustre-OST0000 complete [ 5806.059287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5806.059588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5806.065596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5806.073755] Lustre: Skipped 1 previous similar message [ 5811.171264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5811.181679] Lustre: Skipped 2 previous similar messages [ 5815.273521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5815.287762] Lustre: Skipped 1 previous similar message [ 5817.313405] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5817.460750] Lustre: server umount lustre-MDT0000 complete [ 5820.387427] LustreError: 192192:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5820.402120] LustreError: 192192:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 5821.614600] LustreError: 190929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640843 with bad export cookie 1068062694680929572 [ 5821.623798] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5821.626797] LustreError: 190929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5821.929065] Lustre: server umount lustre-MDT0001 complete [ 5832.667591] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5835.855758] Key type lgssc unregistered [ 5836.255134] LNet: 195839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5836.260974] LNetError: 195839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5836.273514] LNet: Removed LNI 192.168.202.102@tcp [ 5837.124755] Key type .llcrypt unregistered [ 5837.132649] Key type ._llcrypt unregistered [ 5864.700845] Key type ._llcrypt registered [ 5864.702905] Key type .llcrypt registered [ 5864.782340] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5865.854288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5865.921486] alg: No test for adler32 (adler32-zlib) [ 5867.024212] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 5867.186522] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5868.792226] Key type lgssc registered [ 5870.123055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5880.121546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5880.168133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5881.744634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5887.135746] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5892.068436] LustreError: 197253:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5892.084921] LustreError: 197253:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5895.965479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5896.333912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5901.947810] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5910.565508] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5917.189717] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5924.882033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5925.256827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5927.297614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5932.335739] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5941.221544] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5945.348963] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5953.248813] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5966.445277] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5978.594993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5978.596461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5978.607841] LustreError: Skipped 1 previous similar message [ 5978.612318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5978.640116] Lustre: Skipped 1 previous similar message [ 5980.541253] Lustre: server umount lustre-OST0000 complete [ 5988.323717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5988.327443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.333551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5988.335959] Lustre: Skipped 1 previous similar message [ 5990.568561] LustreError: 200354:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000062f80fcf) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5992.685519] Lustre: server umount lustre-MDT0000 complete [ 5993.962058] LustreError: 197254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5993.976539] LustreError: 197254:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5996.569326] LustreError: 197233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641018 with bad export cookie 2465015358073037509 [ 5996.588677] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5996.593765] LustreError: 197233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5996.935470] Lustre: server umount lustre-MDT0001 complete [ 6008.458893] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6011.744390] Key type lgssc unregistered [ 6012.063094] LNet: 201104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6012.067871] LNetError: 201104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6012.097238] LNet: Removed LNI 192.168.202.102@tcp [ 6012.953334] Key type .llcrypt unregistered [ 6012.955360] Key type ._llcrypt unregistered [ 6033.034076] Key type ._llcrypt registered [ 6033.037673] Key type .llcrypt registered [ 6033.190400] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6034.608616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6034.752633] alg: No test for adler32 (adler32-zlib) [ 6035.885882] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 6036.309693] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6037.992698] Key type lgssc registered [ 6039.164183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6047.605884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6047.628316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6049.116372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6055.121975] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6058.470193] LustreError: 202386:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6058.504342] LustreError: 202386:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6063.589445] LustreError: 202385:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6065.623161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6066.190797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6072.099616] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6082.137382] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6088.541777] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6095.993355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6096.396518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6103.542379] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6113.111306] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6118.984713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6124.709435] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6135.133306] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6143.249867] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6149.628468] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6157.800140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.802036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6157.818030] Lustre: Skipped 1 previous similar message [ 6162.919477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.935600] Lustre: Skipped 2 previous similar messages [ 6163.806542] Lustre: server umount lustre-OST0000 complete [ 6169.057778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6169.061710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6169.075319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6173.160053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6173.168291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.599502] Lustre: server umount lustre-MDT0000 complete [ 6178.275328] LustreError: 202386:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.316548] LustreError: 202386:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6178.473197] LustreError: 202365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641200 with bad export cookie 6148432563752213292 [ 6178.482353] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6178.498195] LustreError: 202365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6178.873720] Lustre: server umount lustre-MDT0001 complete [ 6191.009766] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6194.247493] Key type lgssc unregistered [ 6194.548724] LNet: 207120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6194.555642] LNetError: 207120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6194.570294] LNet: Removed LNI 192.168.202.102@tcp [ 6195.583142] Key type .llcrypt unregistered [ 6195.584358] Key type ._llcrypt unregistered [ 6221.768167] Key type ._llcrypt registered [ 6221.772420] Key type .llcrypt registered [ 6221.862165] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6222.877722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6223.059660] alg: No test for adler32 (adler32-zlib) [ 6224.208317] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 6224.449354] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6226.193142] Key type lgssc registered [ 6227.362835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6236.738783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6236.758675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.432792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6244.559337] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6248.943516] LustreError: 208528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6248.970517] LustreError: 208528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6254.054273] LustreError: 208529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6255.950962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6256.304125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6262.772766] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6272.635234] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6279.659786] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6286.607993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6286.979723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.662919] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6301.254885] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6308.468604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6329.319051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6329.321735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.330038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6329.354927] Lustre: Skipped 1 previous similar message [ 6332.921054] Lustre: server umount lustre-OST0000 complete [ 6338.017260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6338.027641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6338.052539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6338.060536] Lustre: Skipped 1 previous similar message [ 6339.554612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6339.555952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.982589] Lustre: server umount lustre-MDT0000 complete [ 6344.675032] LustreError: 208529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.518157] LustreError: 208510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641370 with bad export cookie 9563204559093454313 [ 6348.521987] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6349.104550] Lustre: server umount lustre-MDT0001 complete [ 6361.729941] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6364.501819] Key type lgssc unregistered [ 6364.767032] LNet: 212231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6364.772349] LNetError: 212231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6364.790385] LNet: Removed LNI 192.168.202.102@tcp [ 6365.559610] Key type .llcrypt unregistered [ 6365.562560] Key type ._llcrypt unregistered [ 6388.120088] Key type ._llcrypt registered [ 6388.122528] Key type .llcrypt registered [ 6388.243800] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6390.180243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6390.212652] alg: No test for adler32 (adler32-zlib) [ 6391.311493] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 6391.603552] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6393.280215] Key type lgssc registered [ 6394.677678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6403.208605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6403.223255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6404.736133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6409.576648] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6414.332278] LustreError: 213496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6414.368157] LustreError: 213496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6418.851966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6419.440213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6424.774705] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6434.772166] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6441.067904] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6450.068167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6450.212656] LustreError: 215317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6450.455420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6455.797298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6458.457347] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6466.824131] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6481.382471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6481.395967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6481.423687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6483.425852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6483.428901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6487.268632] Lustre: server umount lustre-OST0000 complete [ 6493.669187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6493.672566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6493.685779] Lustre: Skipped 1 previous similar message [ 6493.715791] Lustre: Skipped 2 previous similar messages [ 6497.807877] Lustre: server umount lustre-MDT0000 complete [ 6498.789845] LustreError: 215991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6498.813675] LustreError: 215991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6502.316764] LustreError: 213475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641524 with bad export cookie 7016171449371771810 [ 6502.324673] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6502.760147] Lustre: server umount lustre-MDT0001 complete [ 6518.115865] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6520.870876] Key type lgssc unregistered [ 6521.035620] LNet: 217183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6521.041562] LNetError: 217183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6522.086966] LNet: Removed LNI 192.168.202.102@tcp [ 6522.911771] Key type .llcrypt unregistered [ 6522.914651] Key type ._llcrypt unregistered [ 6549.258635] Key type ._llcrypt registered [ 6549.260268] Key type .llcrypt registered [ 6549.388763] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6551.563145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6551.647530] alg: No test for adler32 (adler32-zlib) [ 6552.871628] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 6553.285406] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6555.032931] Key type lgssc registered [ 6558.051894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6567.081584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6567.103980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6568.697766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6574.548295] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6579.189617] LustreError: 218445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.215775] LustreError: 218445:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6584.290741] LustreError: 218446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6586.210353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.610841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6592.159202] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6601.929407] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6608.942801] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6616.469259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6616.981178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6624.122978] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6633.193546] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6638.585118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6647.580944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.690641] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6647.930039] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6655.316155] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6664.474158] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6666.375584] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6678.233433] Lustre: Permanently deactivating lustre-OST0001 [ 6678.247432] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6686.827832] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6698.037073] Lustre: Permanently reactivating lustre-OST0001 [ 6698.080343] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6698.102710] Lustre: Skipped 2 previous similar messages [ 6707.932462] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.963491] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6707.985588] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6708.019947] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6708.189092] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6713.315331] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6713.326835] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6713.343916] Lustre: Skipped 1 previous similar message [ 6713.355468] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6713.366195] Lustre: Skipped 1 previous similar message [ 6714.866986] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6719.092657] Lustre: server umount lustre-OST0001 complete [ 6719.975600] LustreError: 220271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6720.026218] LustreError: 220271:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6725.091333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.093463] LustreError: 220270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6725.096332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6725.096341] Lustre: Skipped 1 previous similar message [ 6725.126258] Lustre: Skipped 2 previous similar messages [ 6725.195169] LustreError: 220270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6730.220745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6730.225598] LustreError: 220271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6730.226610] Lustre: Skipped 2 previous similar messages [ 6730.252246] LustreError: 220271:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6735.331382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6735.335471] Lustre: Skipped 1 previous similar message [ 6738.401620] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6738.729685] Lustre: server umount lustre-OST0000 complete [ 6745.572328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6745.585449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6745.595721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6745.595731] LustreError: Skipped 1 previous similar message [ 6745.597428] Lustre: Skipped 1 previous similar message [ 6745.626856] Lustre: Skipped 1 previous similar message [ 6749.030367] Lustre: server umount lustre-MDT0000 complete [ 6750.691878] LustreError: 219174:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.702199] LustreError: 219174:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6753.449932] LustreError: 218425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641775 with bad export cookie 5491194822148392069 [ 6753.456376] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6753.466391] LustreError: 218425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6754.177678] Lustre: server umount lustre-MDT0001 complete [ 6770.925163] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6774.725472] Key type lgssc unregistered [ 6775.158035] LNet: 223544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6775.163926] LNetError: 223544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6775.178050] LNet: Removed LNI 192.168.202.102@tcp [ 6776.725078] Key type .llcrypt unregistered [ 6776.728917] Key type ._llcrypt unregistered [ 6825.416345] Key type ._llcrypt registered [ 6825.419342] Key type .llcrypt registered [ 6825.539935] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6826.842076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6827.013847] alg: No test for adler32 (adler32-zlib) [ 6828.198823] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 6828.405025] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6830.088362] Key type lgssc registered [ 6831.343476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6839.260439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6839.274216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.541332] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6840.568855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6840.774859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6845.088390] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6855.208196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6855.308631] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6855.343270] Lustre: 225747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6855.814622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6862.383837] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6872.347259] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6879.909386] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6891.135293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6891.397374] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6891.856345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6897.238081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6899.038651] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6909.119971] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6913.462501] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6920.963681] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6928.085774] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6937.105400] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6945.265743] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6964.307233] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6964.315223] Lustre: Skipped 2 previous similar messages [ 7004.146685] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7004.153906] Lustre: Skipped 4 previous similar messages [ 7068.610075] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7068.614525] Lustre: Skipped 7 previous similar messages [ 7086.572332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7086.572768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.593984] LustreError: Skipped 1 previous similar message [ 7086.595360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7086.645716] Lustre: Skipped 1 previous similar message [ 7090.865245] Lustre: server umount lustre-OST0000 complete [ 7096.291424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7096.298888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.320712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7096.331978] Lustre: Skipped 1 previous similar message [ 7101.922328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.927663] Lustre: Skipped 1 previous similar message [ 7107.046179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7107.057362] Lustre: Skipped 1 previous similar message [ 7109.088298] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7109.256625] Lustre: server umount lustre-MDT0000 complete [ 7112.841945] LustreError: 225738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642134 with bad export cookie 9074089519925337767 [ 7112.851325] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7112.854199] LustreError: 225738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7113.153766] Lustre: server umount lustre-MDT0001 complete [ 7125.922239] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7129.112322] Key type lgssc unregistered [ 7129.414129] LNet: 230530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7129.432475] LNetError: 230530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7129.457901] LNet: Removed LNI 192.168.202.102@tcp [ 7130.415825] Key type .llcrypt unregistered [ 7130.417741] Key type ._llcrypt unregistered [ 7163.778436] Key type ._llcrypt registered [ 7163.782521] Key type .llcrypt registered [ 7163.992139] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7165.162912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7165.273347] alg: No test for adler32 (adler32-zlib) [ 7166.543892] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 7166.752843] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7168.416236] Key type lgssc registered [ 7169.749351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7178.929566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7178.945892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7180.434040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7185.898795] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7191.012333] LustreError: 231940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7191.046429] LustreError: 231940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7195.983638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7196.133425] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7196.358281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7200.602543] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7210.741102] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7217.924967] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7226.133457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7226.613977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7232.792348] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7241.684335] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7251.196308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7266.792556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7266.808056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7266.817079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7266.836239] Lustre: Skipped 1 previous similar message [ 7266.850281] Lustre: Skipped 1 previous similar message [ 7271.907415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7271.917402] Lustre: Skipped 1 previous similar message [ 7273.028308] Lustre: server umount lustre-OST0000 complete [ 7278.561920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7278.572675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7278.592711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7282.151996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7284.162538] Lustre: server umount lustre-MDT0000 complete [ 7287.265932] LustreError: 231940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7287.300683] LustreError: 231940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7288.998581] LustreError: 231921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642310 with bad export cookie 9814499721376922514 [ 7289.011276] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7289.399434] Lustre: server umount lustre-MDT0001 complete [ 7301.996417] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7304.927532] Key type lgssc unregistered [ 7305.251051] LNet: 235658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7305.259951] LNetError: 235658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7306.284867] LNet: Removed LNI 192.168.202.102@tcp [ 7307.249917] Key type .llcrypt unregistered [ 7307.252244] Key type ._llcrypt unregistered [ 7333.432990] Key type ._llcrypt registered [ 7333.441847] Key type .llcrypt registered [ 7333.642974] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7334.966654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7335.040187] alg: No test for adler32 (adler32-zlib) [ 7336.208706] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 7336.496735] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7338.181089] Key type lgssc registered [ 7339.368882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7608.444361] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7611.834649] Key type lgssc unregistered [ 7612.263845] LNet: 237683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7612.275928] LNetError: 237683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7612.297268] LNet: Removed LNI 192.168.202.102@tcp [ 7613.340477] Key type .llcrypt unregistered [ 7613.343152] Key type ._llcrypt unregistered [ 7641.373659] Key type ._llcrypt registered [ 7641.378475] Key type .llcrypt registered [ 7641.594858] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 7657.284344] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7658.520593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7658.551858] alg: No test for adler32 (adler32-zlib) [ 7659.909276] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 7660.191122] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7661.885161] Key type lgssc registered [ 7663.331662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7697.908677] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7432 sec ========== 06:58:37 (1789642717) [ 7700.578980] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:58:39 (1789642719) === [ 7705.123489] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:58:44 (1789642724) === [ 7755.764074] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7759.778995] Key type lgssc unregistered [ 7760.205295] LNet: 242921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7760.217531] LNetError: 242921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7760.265921] LNet: Removed LNI 192.168.202.102@tcp [ 7761.141384] Key type .llcrypt unregistered [ 7761.142938] Key type ._llcrypt unregistered